Method, device, computer equipment and storage medium for analyzing people in group photo images

Abstract

The present invention relates to the field of computer vision technology, and provides a group photo image character analysis method, device, computer equipment and storage medium, including obtaining a first high-dimensional feature vector of a face image of a person to be analyzed and a second high-dimensional feature vector set of the face image of the person in the group photo; taking the first high-dimensional feature vector as the first cluster center and adding it to a cluster center set; clustering the second high-dimensional feature vector set with the cluster center set to determine similar second high-dimensional feature vectors and similarities of each cluster center; updating each cluster center with similar second high-dimensional feature vectors and selecting new cluster centers according to similarity to add to the cluster center set; returning to clustering until the cluster center set does not change, removing the cluster center most similar to the first high-dimensional feature vector to obtain a final cluster center set; taking the face image corresponding to the final cluster center set as the target person to determine the relationship between the person profile and the person to be analyzed. The method can accurately mine the person association relationship.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method, device, computer equipment and storage medium for analyzing people in a group photo. Background Art

[0002] With the rapid development of Internet technology, the number of public Internet data such as social networks and various website platforms is increasing. Through in-depth mining and application of these big data, a large amount of valuable information can be discovered and associated. Among them, group photos in Internet public data often contain important information about the relationship between people. Analyzing and mining the relationship between people in group photos through massive images is an important way to build social network knowledge graphs and character relationship graphs.

[0003] Currently, the focus of research in academia and industry is on face detection technology and face recognition technology. However, group photo analysis and character mining based on big data focuses more on information fusion under massive data, which is different from face detection and character recognition technology. Therefore, there is still a need for a method that can accurately mine character associations from massive group photo image data. Summary of the invention

[0004] Based on this, it is necessary to provide a group photo character analysis method, device, computer equipment and storage medium that can accurately mine character association relationships for massive group photo image data in order to address the above technical problems.

[0005] The present invention provides a method for analyzing characters in a group photo, comprising:

[0006] Obtain a first high-dimensional feature vector of a facial image of a person to be analyzed and a second high-dimensional feature vector set of a facial image of a person who is photographed with the person to be analyzed;

[0007] Adding the first high-dimensional feature vector as the first cluster center to the cluster center set;

[0008] Clustering the second high-dimensional feature vector set with the cluster center set, and determining similar second high-dimensional feature vectors and similarities of each cluster center;

[0009] Using the similar second high-dimensional feature vectors to update each of the cluster centers respectively, and selecting the second high-dimensional feature vectors as cluster centers according to the similarity to add to the cluster center set;

[0010] Returning to the step of clustering the second high-dimensional feature vector set with the cluster center set, until the cluster center set does not change, removing the cluster center with the highest similarity to the first high-dimensional feature vector, and obtaining a final cluster center set;

[0011] The facial image corresponding to each cluster center in the final cluster center set is taken as the target person, and the person profile of the target person and the closeness of the relationship with the person to be analyzed are determined.

[0012] In one embodiment, the step of obtaining a first high-dimensional feature vector of a facial image of a person to be analyzed and a set of second high-dimensional feature vectors of a facial image of a person who is photographed with the person to be analyzed includes:

[0013] Acquire a facial image of a person to be analyzed, perform facial feature encoding on the facial image, and extract a first high-dimensional feature vector of the facial image;

[0014] Filtering the group photo image to obtain the second highest dimensional feature vector of each face in the group photo image;

[0015] Retrieving and extracting a group photo face set from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector;

[0016] The second highest dimensional feature vectors of each group photo face in the group photo face set are combined as a second high dimensional feature vector set.

[0017] In one embodiment, the retrieving and extracting a group face set of people who are photographed with the person to be analyzed from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector includes:

[0018] Determine the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector using vector cosine similarity;

[0019] Retrieving a candidate second high-dimensional feature vector similar to the first high-dimensional feature vector from the second high-dimensional feature vector using an approximate nearest neighbor search algorithm based on the spatial distance;

[0020] Among the group photo images, group photo images including the candidate second high-dimensional feature vector are screened and face images are extracted to form a group photo face set.

[0021] In one embodiment, the updating of each cluster center respectively using the similar second high-dimensional feature vector comprises:

[0022] Calculating the feature vector means of the similar second high-dimensional feature vectors corresponding to the cluster centers respectively;

[0023] The mean values ​​of the feature vectors are assigned to the corresponding cluster centers respectively.

[0024] In one embodiment, selecting the second high-dimensional feature vector as a cluster center and adding it to the cluster center set according to the similarity includes:

[0025] Obtaining preset clustering hyperparameters and selecting a minimum similarity from the similarities;

[0026] When the minimum similarity is not greater than the clustering hyperparameter, a second high-dimensional feature vector corresponding to the minimum similarity is added to the clustering center set as a clustering center.

[0027] In one embodiment, the method for determining the closeness of the relationship between the target person and the person to be analyzed includes:

[0028] The relationship closeness is calculated according to the number of the second high-dimensional feature vectors, the number of the similar second high-dimensional feature vectors corresponding to the cluster center of the target person, and the variance of the similar second high-dimensional feature vectors.

[0029] In one embodiment, the method for determining the target person's profile includes:

[0030] Calculate the cosine similarity between the target person's face image and each face image in the person archive respectively;

[0031] The character profile corresponding to the face portrait with the highest cosine similarity is determined as the character profile of the target person.

[0032] A device for analyzing characters in a group photo, comprising:

[0033] An acquisition module, used to acquire a first high-dimensional feature vector of a facial image of a person to be analyzed and a set of second high-dimensional feature vectors of a facial image of a person who is photographed with the person to be analyzed;

[0034] A clustering module, used to add the first high-dimensional feature vector as the first cluster center to a cluster center set; cluster the second high-dimensional feature vector set with the cluster center set to determine similar second high-dimensional feature vectors and similarities of each cluster center;

[0035] An updating module, configured to update each of the cluster centers using the similar second high-dimensional feature vectors, and to select the second high-dimensional feature vectors as cluster centers according to the similarity and add them to the cluster center set;

[0036] a removal module, configured to return to the step of clustering the second high-dimensional feature vector set with the cluster center set until the cluster center set does not change, and remove the cluster center with the highest similarity to the first high-dimensional feature vector to obtain a final cluster center set;

[0037] The determination module is used to take the facial image corresponding to each cluster center in the final cluster center set as the target person, and determine the person profile of the target person and the closeness of the relationship with the person to be analyzed.

[0038] The present invention also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the group photo image character analysis method described in any one of the above items are implemented.

[0039] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for analyzing characters in group photo images are implemented.

[0040] The above-mentioned group photo image character analysis method, device, computer equipment and storage medium obtain the first high-dimensional feature vector of the face image of the person to be analyzed and the second high-dimensional feature vector set of the group photo face image, and then use the first high-dimensional feature vector as the first cluster center to join the cluster center set; then cluster the second high-dimensional feature vector set with the cluster center set to determine the similar second high-dimensional feature vectors and similarities of each cluster center; then use the similar second high-dimensional feature vectors to update each cluster center and select the second high-dimensional feature vectors as new cluster centers according to the similarity to join the cluster center set; return to clustering until the cluster center set does not change, then remove the cluster center most similar to the first high-dimensional feature vector to obtain the final cluster center set; use the face image corresponding to the final cluster center set as the target person to determine the character profile, and determine the closeness of the relationship with the person to be analyzed, thereby realizing the mining of the association relationship between people from massive group photo image data, and improving the accuracy of the person association relationship mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 FIG. 1 is a diagram showing an application environment of a method for analyzing characters in a group photo in one embodiment.

[0042] Figure 2 The figure is a flowchart of a method for analyzing characters in a group photo in one embodiment.

[0043] Figure 3 FIG. 1 is a schematic diagram of a facial image of a person to be analyzed in an embodiment.

[0044] Figure 4 Schematic diagram of the first round of clustering analysis in one embodiment.

[0045] Figure 5 Schematic diagram of the second round of clustering analysis in one embodiment.

[0046] Figure 6Schematic diagram of the third round of clustering analysis in one embodiment.

[0047] Figure 7 A schematic diagram of cluster analysis results in one embodiment.

[0048] Figure 8 The figure is a schematic diagram of the relationship between the target person and the person to be analyzed in one embodiment.

[0049] Fig. 9 1 is a structural block diagram of a device for analyzing characters in a group photo in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] The group photo character analysis method provided in this application can be applied to Figure 1 In the application environment shown, the application environment involves a terminal 102 and a server 104. The terminal 102 communicates with the server 104 through a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0052] When the terminal 102 receives the character analysis instruction, the terminal 102 can implement the above group photo character analysis method alone. The terminal 102 can also send the character analysis instruction to the communicating server 104, and the server 104 implements the above group photo character analysis method.

[0053] Taking server 104 as an example, specifically, server 104 obtains the first high-dimensional feature vector of the facial image of the person to be analyzed and the second high-dimensional feature vector set of the facial image of the person to be analyzed; server 104 adds the first high-dimensional feature vector as the first cluster center to the cluster center set; server 104 clusters the second high-dimensional feature vector set with the cluster center set to determine the similar second high-dimensional feature vectors and similarities of each cluster center; server 104 updates each cluster center respectively using the similar second high-dimensional feature vectors, and selects the second high-dimensional feature vectors as cluster centers according to the similarity and adds them to the cluster center set; returns to the step of clustering the second high-dimensional feature vector set with the cluster center set until the cluster center set does not change, and server 104 removes the cluster center with the highest similarity to the first high-dimensional feature vector to obtain the final cluster center set; server 104 uses the facial image corresponding to each cluster center in the final cluster center set as the target person, determines the person profile of the target person and the closeness of the relationship with the person to be analyzed.

[0054] In one embodiment, Figure 2 As shown, a method for analyzing characters in a group photo is provided, and the method is applied to a server as an example for explanation, and includes the following steps:

[0055] Step S201, obtaining a first high-dimensional feature vector of a facial image of a person to be analyzed and a set of second high-dimensional feature vectors of a facial image of a person who is photographed with the person to be analyzed.

[0056] Specifically, after receiving the character analysis instruction, the server first obtains the facial image of the person to be analyzed, and then uses the facial feature encoding technology to convert the facial image of the person to be analyzed into a high-dimensional feature vector, thereby obtaining the first high-dimensional feature vector in this embodiment. At the same time, the server further obtains the facial image of the person who has taken a group photo with the person to be analyzed from the group photo image, and also uses the facial feature encoding technology to obtain the high-dimensional feature vector of the group photo facial image, thereby obtaining the second high-dimensional feature vector in this embodiment. The server further combines the second high-dimensional feature vectors of all the group photo facial images into a second high-dimensional feature vector set. Figure 3 As shown, a schematic diagram of a face image of a person to be analyzed is provided, wherein: Figure 3 Figure (a) is an actual facial image, and the rectangle in Figure (b) is used to replace the high-dimensional feature vector of the facial image of the person to be analyzed for the convenience of subsequent illustrations. It can be understood that the rectangles appearing in the figures all represent the high-dimensional feature vector of the facial image of the person to be analyzed.

[0057] In one embodiment, step S201 includes: obtaining a facial image of a person to be analyzed, performing facial feature encoding on the facial image to extract a first high-dimensional feature vector of the facial image; screening group photos to obtain a second high-dimensional feature vector of each face in the group photo; retrieving and extracting a group photo face set from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector; and combining the second high-dimensional feature vectors of each group photo face set as a second high-dimensional feature vector set.

[0058] Specifically, the server obtains the face image X of the person to be analyzed input by the system, and uses the face detection technology to extract the face sub-image in the face image X. If there are multiple face sub-images, the server specifies the face sub-image that the user is interested in as the final face image f of the person to be analyzed by interacting with the user. X The server then processes the face image f X Use ArcFace facial feature encoding technology to get the first high-dimensional feature vector v X Then, based on the filtering condition, the server selects group photos from the constructed face image base library M and obtains the second highest dimensional feature vector of each face image in the group photos. Since it is a group photo image, the filtering condition can be an image with a number of faces greater than or equal to 2.

[0059] The server further calculates the similarity between the first high-dimensional feature vector and each second high-dimensional feature vector, and quickly retrieves group photos that are highly similar to the first high-dimensional feature vector from the group photos, and regards each selected group photo image as a group photo image set including the person to be analyzed, that is, the group photos in the group photo image set are all group photos taken with the person to be analyzed. In the process of selecting the group photo image set, the server can sort them by similarity, and preferably form a set of the TOP K group photos with the closest similarity. Finally, for each group photo image in the selected group photo image set, extract a facial image therefrom as a group photo face set Fsub that has been taken with the person to be analyzed, Fsub={f1,f2……f n}, and the second highest dimensional feature vector combination of each face image in the group face set Fsub is used as the second highest dimensional feature vector set Vsub for the group photo with the person to be analyzed, Vsub = {v1, v2...v n}, there is a one-to-one correspondence between the elements in Fsub and Vsub.

[0060] The face image base M can be pre-built, for example, a large amount of public image data S = {I1, I2...I n} as the face image base M. The second high-dimensional feature vector of each face image in the group photo image in the face image base M can be obtained by using the face feature encoding technology when constructing the face image base M, and then stored together after being associated with the corresponding group photo image. After the subsequent server obtains the group photo image from the face image base M, it can directly obtain the high-dimensional feature vector of each face image in the group photo image based on the association. Specifically: the face sub-images f in the image data I are obtained in turn by using the face detection technology I , ArcFace facial feature encoding technology is also used to encode the face sub-image f of the image data I Transformed into a high-dimensional feature vector v I After that, the image information of the image data and its corresponding face sub-images are associated with the high-dimensional feature vectors and stored in a database, and an index is created with the high-dimensional feature vectors. The database in this embodiment preferably uses Milvus vector data.

[0061] Step S202: adding the first high-dimensional feature vector as the first cluster center to the cluster center set.

[0062] Specifically, after obtaining the high-dimensional feature vector, the server adds the first high-dimensional feature vector as the first cluster center for clustering to the cluster center set C, that is, the first element c1=v in C X , c1 is Figure 3 As shown in (b), at this time the set C = {c1}.

[0063] Step S203: clustering the second high-dimensional feature vector set with the cluster center set, and determining similar second high-dimensional feature vectors and similarities of each cluster center.

[0064] Specifically, the second highest dimensional feature vector set Vsub = {v1, v2...v n}, calculate each v separately n and any cluster center c in the cluster center set C n The category of the cluster center with the highest similarity is selected as v n The category to which v belongs, that is, n The cluster center is the most similar, and is the second highest dimensional feature vector of the cluster center. At the same time, the corresponding similarity is recorded. The similarity can be calculated by any existing similarity calculation method. In this embodiment, the cosine similarity is preferably used for similarity calculation, which is recorded as sim(v n , c n ), all similarities form a set SIMsub. Figure 4 As shown in Figure 1, a schematic diagram of the first round of cluster analysis is provided. Figure 4The rectangle in is the first high-dimensional feature vector of the person to be analyzed, which is used to replace the facial image of the person to be analyzed. The triangle and circle are the second high-dimensional feature vectors of the face in the group photo image, which are used to replace the facial image of the person to be analyzed. The same shape with different shades of color is used to represent the facial state of the same face in different images, such as faces at different angles. Figure 4 (a) is the first round of clustering results. Figure 4 (b) is the newly added cluster center in the first round. This is the first round of clustering, so the cluster center is c1.

[0065] Step S204: using similar second high-dimensional feature vectors to update each cluster center respectively, and selecting second high-dimensional feature vectors as cluster centers according to similarity to add to the cluster center set.

[0066] Step S205, returning to step S203, until the set of cluster centers does not change, remove the cluster center with the highest similarity to the first high-dimensional feature vector to obtain a final set of cluster centers.

[0067] Specifically, after completing a round of clustering, the server updates the cluster center set based on the clustering results. First, since the image data collected from the Internet cannot ensure that the faces are all at standard angles, the faces may be at multiple angles in the image data. Therefore, in order to ensure the accuracy of the cluster center, the cluster center is updated based on each similar second-highest dimensional feature vector similar to the cluster center. Subsequent iterative clustering based on the updated cluster center can improve the accuracy of clustering. At the same time, based on the similarity of all similar second-highest dimensional feature vectors, a second-highest dimensional feature vector is selected as a new cluster center and added to the cluster center set C.

[0068] If the number of cluster centers in the cluster center set increases, or any of the cluster centers is updated, it means that the cluster center set has changed, which is equivalent to generating new cluster centers. At this time, the next round of clustering should be continued, so return to step S203 and perform clustering again with the updated cluster centers. Figure 5 and Figure 6 As shown, Figure 5 A schematic diagram of the second round of cluster analysis is provided. Figure 5 (a) is the second round of clustering results. Figure 5 (b) is the newly added cluster center in the second round of clustering. Figure 6 A schematic diagram of the third round of cluster analysis is provided. Figure 6 (a) is the third round of clustering results. Figure 6(b) is the newly added cluster center in the third round of clustering. That is, in the first round of clustering, the cluster center set C = {c1}, in the second round of clustering, the cluster center set C = {c1, c2}, and in the third round of clustering, the cluster center set C = {c1, c2, c3}. In addition, it should be understood that in the next round of clustering, the cluster center existing in the previous round may be updated. Assuming c1 is taken as an example, when c1 is first updated to c 11 , then the second round of clustering center set C = {c 11 , c2}, if it is updated to c for the second time 111 , the third round of clustering center set C = {c 111 , c2, c3}. Reference Figure 4-6 It can be seen that after multiple clustering, the clustering division of each face image becomes more and more accurate.

[0069] If the number of cluster centers in the cluster center set does not increase and all cluster centers are not updated, indicating that the cluster center aggregation has not changed, it is considered that the most accurate cluster center has been generated, that is, it is considered that all the people appearing in the group photo face set have been enumerated, and clustering is stopped at this time. After stopping clustering, the current latest cluster center set is obtained, which includes the person to be analyzed. Therefore, the cluster center with the highest similarity to the first high-dimensional feature vector is removed from the current latest cluster center set to obtain the final cluster center set. For example, the latest cluster center set C after clustering is stopped f ={c1, c2, ..., c m , c m+1}, calculate C in turn f Any cluster center and the first high-dimensional feature vector v X The cosine similarity of , select the cluster center C with the highest similarity f Remove it from it and get the final cluster center set C e ={c1, c2, ..., c m}.

[0070] Step S206, taking the facial image corresponding to each cluster center in the final cluster center set as the target person, determining the person profile of the target person and the closeness of the relationship with the person to be analyzed.

[0071] Specifically, since the final cluster center set is regarded as the optimal value obtained after all the characters appearing in the group photo face set have been enumerated, the facial image corresponding to each cluster center in the final cluster center set is the target person who has a certain correlation with the person to be analyzed. Therefore, the person corresponding to the facial image corresponding to each cluster center in the final cluster center set is taken as the target person. Then, based on Internet data, such as public information such as social networks, relevant information of the target person is searched to establish his personal profile. At the same time, the closeness of the relationship between the target person and the person to be analyzed is determined based on the number of all group photo faces and the number of faces in the category to which each cluster center belongs.

[0072] In one embodiment, the method for determining the relationship closeness between the target person and the person to be analyzed in step S206 includes: calculating the relationship closeness based on the number of second high dimensional feature vectors, the number of similar second high dimensional feature vectors corresponding to the cluster center of the target person, and the variance of the similar second high dimensional feature vectors.

[0073] Specifically, the formula for calculating the relationship closeness is as follows:

[0074]

[0075] Among them, len(*) is used to count the number of elements in the collection, Vc m is the cluster center c m Similar similar second highest dimensional feature vector, that is, cluster center c m The face of the category, len(Vc m ) is the cluster center c m The number of faces in the same category. len(Vsub) is the number of elements in the second highest dimensional feature vector set, which is the number of faces in all group photos. std(*) is used to calculate the variance of the elements in the set, so std(Vc m ) is the cluster center c m The variance of similar second-highest-dimensional eigenvectors.

[0076] In addition, for the convenience of viewing, it is possible to further calculate the absolute value of the tightness Rm (C X ) Normalize the relationship density value Rm corresponding to each cluster center. Figure 7 As shown in FIG. 1 , a schematic diagram of cluster analysis results is provided. The normalization of the relationship closeness value Rm is to convert the relationship closeness value Rm of each cluster center to the absolute value Rm of the closeness (C X ) to obtain the normalized relationship closeness value. The normalized relationship closeness value set corresponding to the final cluster center set is recorded as RN = {rn1, rn2 ... rn i}, rni The calculation formula is as follows, where c i Refers to the cluster center in the final set of cluster centers.

[0077]

[0078] In one embodiment, a method for determining a character profile of a target person includes: calculating cosine similarities between the target person's facial image and each facial avatar in a character profile library; and determining the character profile corresponding to the facial avatar with the highest cosine similarity as the character profile of the target person.

[0079] Specifically, the final cluster center set C e The person corresponding to each cluster center is regarded as a target person p m , the closeness of the relationship between the target person and the person to be analyzed is rn m , m={1,2,3……,i}, thus forming the target person set P={(p1|c1,rn1),(p2|c2,rn2),……,(p m |c m ,rn m )}. When there is no profile of the target person in the social network, a profile library can be dynamically constructed in advance. For example, through interaction, the user can enter the name and profile picture of the person on the interface to create a basic profile of the person. Then, based on the existence of the constructed profile library, ArcFace is used to extract the corresponding high-dimensional feature vector for the profile picture in the profile, and the target person p in the target person set P is identified based on the high-dimensional feature vector. m , calculate the cosine similarity between it and all the head portraits in the character archive, and select the head portrait with the highest cosine similarity and a similarity greater than the set threshold as the target person p m Then, you can establish a relationship between the target person and the person to be analyzed in the person archive, such as Figure 8 shown.

[0080] The above-mentioned group photo image character analysis method obtains the first high-dimensional feature vector of the face image of the person to be analyzed and the second high-dimensional feature vector set of the group photo face image, and then uses the first high-dimensional feature vector as the first cluster center to join the cluster center set; then clusters the second high-dimensional feature vector set with the cluster center set to determine the similar second high-dimensional feature vectors and similarities of each cluster center; then uses the similar second high-dimensional feature vectors to update each cluster center and selects the second high-dimensional feature vectors as new cluster centers according to the similarity to join the cluster center set; returns to clustering until the cluster center set does not change, then removes the cluster center that is most similar to the first high-dimensional feature vector to obtain the final cluster center set; uses the face image corresponding to the final cluster center set as the target person to determine the character profile, and determines the closeness of the relationship with the person to be analyzed, thereby realizing the mining of the association relationship between people from massive group photo image data, and improving the accuracy of the person association relationship mining.

[0081] In one embodiment, based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector, a group face set of the person to be analyzed is retrieved and extracted from the group photo image, including: using vector cosine similarity to determine the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector; based on the spatial distance, using an approximate nearest neighbor search algorithm to retrieve a candidate second high-dimensional feature vector similar to the first high-dimensional feature vector from the second high-dimensional feature vector; in the group photo image, screening the group photo images including the candidate second high-dimensional feature vector and extracting facial images to form the group face set.

[0082] Specifically, the similarity between high-dimensional feature vectors, this embodiment adopts an approximate nearest neighbor search algorithm based on vector cosine similarity. First, the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector is calculated using vector cosine similarity, and based on the spatial distance, the approximate nearest neighbor search algorithm is used to screen out the TOP K second high-dimensional feature vectors closest to the first high-dimensional feature vector as candidate second high-dimensional feature vectors. Images including candidate second high-dimensional feature vectors are screened out from the group photo images and regarded as a group photo image set of the person to be analyzed. Then, the facial images in each group photo image set are extracted to form a group photo face set.

[0083] In one embodiment, the use of similar second high-dimensional feature vectors in step S204 to update each cluster center respectively includes: calculating the feature vector means of each cluster center corresponding to the similar second high-dimensional feature vector respectively; and assigning each feature vector mean to the corresponding cluster center respectively.

[0084] Specifically, the cluster center and the similar second-highest dimensional feature vectors similar to the cluster center are generally feature vectors of the same face at different facial angles. Therefore, the cluster center is updated in the form of a mean to ensure the accuracy of the cluster center as much as possible. That is, for each cluster center, all the similar second-highest dimensional feature vectors corresponding to it are obtained, the mean of these similar second-highest dimensional feature vectors is calculated, the feature vector mean is obtained, and then the feature vector mean is assigned to the corresponding cluster center to complete the update. For example, assuming that the similar second-highest dimensional feature vectors of c1 include a, b, and c, then the mean of a, b, and c is calculated and assigned to c1 to obtain c. 11 .

[0085] In one embodiment, the step S204 of selecting the second high-dimensional feature vector according to the similarity as the cluster center to be added to the cluster center set includes: obtaining a preset clustering hyperparameter and selecting the minimum similarity from the similarity; when the minimum similarity is not greater than the clustering hyperparameter, adding the second high-dimensional feature vector corresponding to the minimum similarity as the cluster center to the cluster center set.

[0086] Specifically, when the number of cluster center sets is updated, the preset clustering hyperparameter T is first obtained. This hyperparameter is a value set based on clustering requirements and clustering test experience to measure whether the clustering is complete. The clustering hyperparameter is compared with the minimum similarity min(SIMsub) in the similarity set SIMsub. When min(SIMsub) is greater than T, the cluster center is considered to be generated. If min(SIMsub) is not greater than, that is, min(SIMsub) is less than or equal to T, it is determined that a new cluster center needs to be added. Then the second highest dimensional feature vector v corresponding to min(SIMsub) is added to the end of the cluster center set as a new cluster center, which is equivalent to discovering a new target person.

[0087] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0088] In one embodiment, Fig. 9As shown, a device for analyzing characters in a group photo is provided, comprising:

[0089] An acquisition module 901 is used to acquire a first high-dimensional feature vector of a facial image of a person to be analyzed and a set of second high-dimensional feature vectors of a facial image of a person who is photographed with the person to be analyzed;

[0090] Clustering module 902, used to add the first high-dimensional feature vector as the first cluster center to the cluster center set; cluster the second high-dimensional feature vector set with the cluster center set, and determine similar second high-dimensional feature vectors and similarities of each cluster center;

[0091] An updating module 903 is used to update each cluster center by using a similar second high-dimensional feature vector, and select a second high-dimensional feature vector as a cluster center according to the similarity and add it to the cluster center set;

[0092] A removal module 904 is used to return to the step of clustering the second high-dimensional feature vector set with the cluster center set until the cluster center set does not change, and remove the cluster center with the highest similarity to the first high-dimensional feature vector to obtain a final cluster center set;

[0093] The determination module 905 is used to take the facial image corresponding to each cluster center in the final cluster center set as the target person, and determine the person profile of the target person and the closeness of the relationship with the person to be analyzed.

[0094] In one embodiment, the acquisition module 901 is also used to obtain a facial image of a person to be analyzed, perform facial feature encoding on the facial image to extract a first high-dimensional feature vector of the facial image; screen group photos to obtain a second high-dimensional feature vector of each face in the group photo; retrieve and extract a group photo face set from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector; and combine the second high-dimensional feature vectors of each group photo face set as a second high-dimensional feature vector set.

[0095] In one embodiment, the acquisition module 901 is also used to determine the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector using vector cosine similarity; retrieve candidate second high-dimensional feature vectors similar to the first high-dimensional feature vector from the second high-dimensional feature vector based on the spatial distance using an approximate nearest neighbor search algorithm; and in group photos, screen group photos including candidate second high-dimensional feature vectors and extract facial images to form a group face set.

[0096] In one embodiment, the updating module 903 is further used to respectively calculate the feature vector means of similar second high-dimensional feature vectors corresponding to each cluster center; and assign each feature vector mean to the corresponding cluster center.

[0097] In one embodiment, the updating module 903 is also used to obtain preset clustering hyperparameters and filter the minimum similarity from the similarities; when the minimum similarity is not greater than the clustering hyperparameters, the second high-dimensional feature vector corresponding to the minimum similarity is added as the clustering center to the clustering center set.

[0098] In one embodiment, the determination module 905 is further used to calculate the relationship closeness according to the number of second high-dimensional feature vectors, the number of similar second high-dimensional feature vectors corresponding to the cluster center of the target person, and the variance of the similar second high-dimensional feature vectors.

[0099] In one embodiment, the determination module 905 is further used to respectively calculate the cosine similarity between the target person's facial image and each facial avatar in the person archive; and determine the person archive corresponding to the facial avatar with the highest cosine similarity as the person archive of the target person.

[0100] For the specific definition of the group photo image character analysis device, please refer to the definition of the group photo image character analysis method above, which will not be repeated here. Each module in the above-mentioned group photo image character analysis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned group photo image character analysis method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form.

[0101] In one embodiment, a computer device is provided, which may be a server, including a processor, a memory and a network interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for analyzing characters in a group photo image is implemented. Exemplarily, the computer program can be divided into one or more modules, one or more modules are stored in the memory, and are executed by the processor to complete the present invention. One or more modules can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0102] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0103] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0104] Those skilled in the art will appreciate that the computer device structure shown in this embodiment is merely a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the present invention is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0105] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0106] Obtain a first high-dimensional feature vector of a facial image of a person to be analyzed and a second high-dimensional feature vector set of a facial image of a person who is photographed with the person to be analyzed;

[0107] The first high-dimensional feature vector is added as the first cluster center to the cluster center set; the second high-dimensional feature vector set is clustered with the cluster center set to determine similar second high-dimensional feature vectors and similarities of each cluster center;

[0108] Using similar second-highest-dimensional feature vectors to update each cluster center, and selecting the second-highest-dimensional feature vector as a cluster center according to the similarity to add to the cluster center set;

[0109] Return to the step of clustering the second high-dimensional feature vector set with the cluster center set until the cluster center set does not change, and remove the cluster center with the highest similarity to the first high-dimensional feature vector to obtain the final cluster center set;

[0110] The facial image corresponding to each cluster center in the final cluster center set is taken as the target person, and the character profile of the target person and the closeness of the relationship with the person to be analyzed are determined.

[0111] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining a facial image of a person to be analyzed, performing facial feature encoding on the facial image to extract a first high-dimensional feature vector of the facial image; screening group photos to obtain a second high-dimensional feature vector of each face in the group photo image; based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector, retrieving and extracting a group photo face set from the group photo image; and combining the second high-dimensional feature vectors of each group photo face set as a second high-dimensional feature vector set.

[0112] In one embodiment, when the processor executes the computer program, the following steps are also implemented: using vector cosine similarity to determine the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector; based on the spatial distance, using an approximate nearest neighbor search algorithm to retrieve a candidate second high-dimensional feature vector similar to the first high-dimensional feature vector from the second high-dimensional feature vector; in group photos, screening group photos including the candidate second high-dimensional feature vector and extracting facial images to form a group photo face set.

[0113] In one embodiment, when the processor executes the computer program, the following steps are also implemented: respectively calculating the feature vector means of similar second high-dimensional feature vectors corresponding to each cluster center; and assigning each feature vector mean to the corresponding cluster center.

[0114] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining preset clustering hyperparameters and screening the minimum similarity from the similarities; when the minimum similarity is not greater than the clustering hyperparameters, adding the second high-dimensional feature vector corresponding to the minimum similarity as the clustering center to the clustering center set.

[0115] In one embodiment, when the processor executes the computer program, the following steps are also implemented: calculating the relationship closeness based on the number of second high-dimensional feature vectors, the number of similar second high-dimensional feature vectors corresponding to the cluster center of the target person, and the variance of the similar second high-dimensional feature vectors.

[0116] In one embodiment, when the processor executes the computer program, the following steps are also implemented: respectively calculating the cosine similarity between the target person's facial image and each facial avatar in the person archive; determining the person archive corresponding to the facial avatar with the highest cosine similarity as the person archive of the target person.

[0117] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0118] Obtain a first high-dimensional feature vector of a facial image of a person to be analyzed and a second high-dimensional feature vector set of a facial image of a person who is photographed with the person to be analyzed;

[0119] The first high-dimensional feature vector is added as the first cluster center to the cluster center set; the second high-dimensional feature vector set is clustered with the cluster center set to determine similar second high-dimensional feature vectors and similarities of each cluster center;

[0120] Using similar second-highest-dimensional feature vectors to update each cluster center, and selecting the second-highest-dimensional feature vector as a cluster center according to the similarity to add to the cluster center set;

[0121] Return to the step of clustering the second high-dimensional feature vector set with the cluster center set until the cluster center set does not change, and remove the cluster center with the highest similarity to the first high-dimensional feature vector to obtain the final cluster center set;

[0122] The facial image corresponding to each cluster center in the final cluster center set is taken as the target person, and the character profile of the target person and the closeness of the relationship with the person to be analyzed are determined.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a facial image of a person to be analyzed, performing facial feature encoding on the facial image to extract a first high-dimensional feature vector of the facial image; screening group photos to obtain a second high-dimensional feature vector of each face in the group photo image; retrieving and extracting a group photo face set from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector; and combining the second high-dimensional feature vectors of each group photo face set as a second high-dimensional feature vector set.

[0124] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: using vector cosine similarity to determine the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector; based on the spatial distance, using an approximate nearest neighbor search algorithm to retrieve candidate second high-dimensional feature vectors similar to the first high-dimensional feature vector from the second high-dimensional feature vector; in group photos, screening group images including candidate second high-dimensional feature vectors and extracting facial images to form a group face set.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: respectively calculating the feature vector means of similar second high-dimensional feature vectors corresponding to each cluster center; and assigning each feature vector mean to the corresponding cluster center.

[0126] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining preset clustering hyperparameters and screening the minimum similarity from the similarities; when the minimum similarity is not greater than the clustering hyperparameters, adding the second high-dimensional feature vector corresponding to the minimum similarity as the clustering center to the clustering center set.

[0127] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: calculating the relationship closeness based on the number of second high-dimensional feature vectors, the number of similar second high-dimensional feature vectors corresponding to the cluster center of the target person, and the variance of the similar second high-dimensional feature vectors.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: respectively calculating the cosine similarity between the target person's facial image and each facial avatar in the person archive; determining the person archive corresponding to the facial avatar with the highest cosine similarity as the person archive of the target person.

[0129] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for analyzing people in group photo images, characterized in that: include: Obtain a first high-dimensional feature vector of a facial image of a person to be analyzed and a second high-dimensional feature vector set of a facial image of a person who is photographed with the person to be analyzed; Adding the first high-dimensional feature vector as the first cluster center to the cluster center set; Clustering the second high-dimensional feature vector set with the cluster center set, and determining similar second high-dimensional feature vectors and similarities of each cluster center; Using the similar second high-dimensional feature vectors to update each of the cluster centers respectively, and selecting the second high-dimensional feature vectors as cluster centers according to the similarity to add to the cluster center set; Returning to the step of clustering the second high-dimensional feature vector set with the cluster center set, until the cluster center set does not change, removing the cluster center with the highest similarity to the first high-dimensional feature vector, and obtaining a final cluster center set; The facial image corresponding to each cluster center in the final cluster center set is taken as the target person, and the person profile of the target person and the closeness of the relationship with the person to be analyzed are determined.

2. The method according to claim 1, characterized in that The step of obtaining a first high-dimensional feature vector of a facial image of a person to be analyzed and a set of second high-dimensional feature vectors of a facial image of a person who is photographed with the person to be analyzed includes: Acquire a facial image of a person to be analyzed, perform facial feature encoding on the facial image, and extract a first high-dimensional feature vector of the facial image; Filtering the group photo image to obtain the second highest dimensional feature vector of each face in the group photo image; Retrieving and extracting a group photo face set from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector; The second highest dimensional feature vector combination of each group photo face in the group photo face set is taken as a second high dimensional feature vector set.

3. The method according to claim 2, characterized in that The retrieving and extracting a group face set of the person to be analyzed who is in a group photo from the group photo image based on the similarity between the first high-dimensional feature vector and the second high-dimensional feature vector comprises: Determine the spatial distance between the first high-dimensional feature vector and the second high-dimensional feature vector using vector cosine similarity; Retrieving a candidate second high-dimensional feature vector similar to the first high-dimensional feature vector from the second high-dimensional feature vector using an approximate nearest neighbor search algorithm based on the spatial distance; Among the group photo images, group photo images including the candidate second high-dimensional feature vector are screened and face images are extracted to form a group photo face set.

4. The method according to claim 1, characterized in that: The using the similar second high-dimensional feature vectors to respectively update the cluster centers comprises: Calculating the feature vector means of the similar second high-dimensional feature vectors corresponding to the cluster centers respectively; The mean values ​​of the feature vectors are assigned to the corresponding cluster centers respectively.

5. The method according to claim 1, characterized in that The selecting the second high-dimensional feature vector according to the similarity as a cluster center and adding it to the cluster center set includes: Obtaining preset clustering hyperparameters and selecting a minimum similarity from the similarities; When the minimum similarity is not greater than the clustering hyperparameter, a second high-dimensional feature vector corresponding to the minimum similarity is added to the clustering center set as a clustering center.

6. The method according to claim 1, characterized in that The method for determining the closeness of the relationship between the target person and the person to be analyzed includes: The relationship closeness is calculated according to the number of the second high-dimensional feature vectors, the number of the similar second high-dimensional feature vectors corresponding to the cluster center of the target person, and the variance of the similar second high-dimensional feature vectors.

7. The method according to claim 1, characterized in that The method for determining the target person's profile includes: Calculate the cosine similarity between the target person's face image and each face image in the person archive respectively; The character profile corresponding to the face portrait with the highest cosine similarity is determined as the character profile of the target person.

8. A device for analyzing people in group photo images, characterized in that: include: An acquisition module, used to acquire a first high-dimensional feature vector of a facial image of a person to be analyzed and a set of second high-dimensional feature vectors of a facial image of a person who is photographed with the person to be analyzed; A clustering module, used to add the first high-dimensional feature vector as the first cluster center to a cluster center set; cluster the second high-dimensional feature vector set with the cluster center set to determine similar second high-dimensional feature vectors and similarities of each cluster center; An updating module, configured to update each of the cluster centers using the similar second high-dimensional feature vectors, and to select the second high-dimensional feature vectors as cluster centers according to the similarity and add them to the cluster center set; a removal module, configured to return to the step of clustering the second high-dimensional feature vector set with the cluster center set until the cluster center set does not change, and remove the cluster center with the highest similarity to the first high-dimensional feature vector to obtain a final cluster center set; The determination module is used to take the facial image corresponding to each cluster center in the final cluster center set as the target person, and determine the person profile of the target person and the closeness of the relationship with the person to be analyzed.

9. A computer device comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: The processor is used to implement the group photo image character analysis method described in any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for analyzing characters in a group photo image described in any one of claims 1 to 7 is implemented.

Images