Photo album generation method and electronic equipment
By identifying the social circle of the characters in the image and generating albums based on shooting time and location information, the problem of mismatch between the album and the actual situation of the user in the prior art is solved, and the user experience and applicability are improved.
Patent Information
- Application Number
- CN202310555931.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
The photo albums generated by the prior art do not fit enough with the actual social situation of the user, and the user experience needs to be improved.
By identifying the social circles of the characters in the image, generating albums related to the social circle, using heterogeneous images of the characters to determine the intimacy between characters, identify the social circle, and generating albums based on the social circle, combining the shooting time and location information of the image .
The generated album is more in line with the actual social situation of the user, improves the user experience, and does not require user annotation, which is highly intelligent, has strong applicability, and has strong targeted identification results.
Smart Images

Figure CN119003806A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of electronic technology, and in particular to a photo album generation method and electronic equipment. Background Art
[0002] With the rapid development of camera configurations in electronic devices such as mobile phones and tablet computers, taking photos through electronic devices has become one of the important ways for people to record their lives. And with the rapid development of storage capacity and network technology of electronic devices, the number of images stored in the gallery of user electronic devices is also increasing. Gallery applications (applications, APPs) of some electronic devices can summarize and classify the images stored in the gallery, select some images to generate albums with specific themes, thereby bringing a better experience to users.
[0003] In the related art, gallery apps mainly generate albums with corresponding themes based on the shooting time of the images or the frequency of appearance of people in the images. For example, select images in the gallery that were shot on weekends to generate an album with the theme of "Weekend Time"; for another example, select images with people that appear frequently in the gallery to generate an album with the theme of "Portrait Selection".
[0004] However, the photo album generated by the above method is not sufficiently consistent with the user's actual situation, and the user experience needs to be improved. Summary of the invention
[0005] The present application provides a photo album generation method and electronic device. The generated photo album is related to the social circle of the person in the image, so the generated photo album is closer to the user's actual social situation, improving the user experience.
[0006] In a first aspect, the present application provides a method for generating an album, which is executed by an electronic device and includes:
[0007] A character image heterogeneous map is determined based on multiple images in a gallery application, and the character image heterogeneous map represents the correspondence between the multiple images and the multiple characters, wherein, among the multiple images, an image including a first character corresponds to the first character, and the first character is any one of the multiple characters; based on the character image heterogeneous map, the intimacy between every two characters in the multiple characters is determined, and the intimacy represents the degree of association between every two characters; based on the intimacy between every two characters and the character image heterogeneous map, a first social circle is identified, and the first social circle is a collection of characters with social relationships; a first album is generated based on the first social circle, and the first album includes images in the first circle, and the images in the first circle are images including at least one character in the first social circle.
[0008] In other words, the character heterogeneous graph represents the correspondence between multiple images and the characters included in each of the multiple images. For example, if image A includes character a, then image A corresponds to character a. It can be understood that each image can correspond to multiple characters, and each character can also correspond to multiple images.
[0009] The intimacy between two characters represents the intimacy of the social relationship between the two characters. The intimacy between two characters is a value greater than or equal to 0. The greater the intimacy, the closer the two characters are. If the intimacy is 0, it means that there is no social relationship between the two characters.
[0010] It can be understood that one or more social circles can be identified based on the intimacy between every two characters, and the first social circle can be any one of them. It is used for illustration here and is not used to specifically refer to a certain social circle.
[0011] Optionally, in the process of generating the first album according to the first social circle, it is also possible to further combine the shooting time information of the image, the shooting location of the image, the intimacy, etc. The embodiment of the present application does not impose any limitation on this.
[0012] The album generation method provided in the first aspect determines a character image isomeric map based on multiple images in a gallery application, determines the intimacy between each two characters in a plurality of characters based on the character image isomeric map, and then identifies the first social circle based on the intimacy between each two characters and the character image isomeric map, and then generates the first album based on the first social circle. In this way, the generated first album is related to the social circle of the characters in the image, which will be more in line with the user's actual social situation and improve the user experience. Moreover, in the above-mentioned album generation method, the social circle can be identified based on multiple images in the gallery application without relying on the user's annotation, which is highly intelligent and further improves the user experience. In addition, the above-mentioned process can be performed based on images stored in the electronic device, so it can be completed offline without relying on the network. Compared with identifying social circles through social behaviors on social networks, this method is more applicable, and the recognition results are more targeted for users of electronic devices, thereby further improving the user experience.
[0013] In one possible implementation, the character image heterogeneous graph includes multiple image nodes and multiple character nodes, the multiple image nodes correspond one-to-one to the multiple images, and the multiple character nodes correspond one-to-one to the multiple characters; according to the intimacy between every two characters and the character image heterogeneous graph, the first social circle is identified, including: removing the multiple image nodes in the character image heterogeneous graph, and according to the intimacy between every two characters, connecting the character nodes corresponding to two characters whose intimacy is not 0 through edges to obtain a character connection graph; according to the character connection graph, the first social circle is identified.
[0014] In the character connection graph, the weight of each edge is the intimacy between the characters corresponding to the two character nodes connected by the edge.
[0015] In this implementation, the character image heterogeneous graph is simplified into a character connection graph, and there is no need to find the relationship between characters based on the images in the gallery application. This simplifies the process of obtaining the character connection graph and improves the algorithm operation efficiency.
[0016] Optionally, the character connection graph may not include node information, but only include nodes and edges, as well as edge weights (i.e., intimacy). This can further simplify the structure of the character connection graph and improve the algorithm operation efficiency.
[0017] In one possible implementation, a first social circle is identified based on a character connection graph, including: identifying a central character based on a character image heterogeneous graph, wherein the time distribution divergence and location distribution divergence of the image corresponding to the central character meet a first preset condition, the time distribution divergence is the distribution divergence of the shooting time of the image, and the location distribution divergence is the distribution divergence of the shooting location of the image; removing the character node corresponding to the central character in the character connection graph to obtain a decentralized character connection graph; and performing community discovery on the decentralized character connection graph to obtain the first social circle.
[0018] Generally speaking, in the character connection graph, there are many edges connecting the central character node to other nodes, and the central character has a high degree of intimacy with other characters. Therefore, the weight of the edge connected to the central character node will be very high. If the central character node is included in the community discovery, it is easy to divide the central character node into a certain community in a certain round of iteration. In this way, in the next round of iterations, when the community where the central character node is located is used as a node, there are many edges with other nodes, and / or the sum of the weights of the edges pointing to the community where the central character node is located will be very high, so other points that do not belong to the community will be divided into the community, resulting in inaccurate social circles generated in the end. Therefore, in this implementation method, community discovery is performed based on the decentralized central character connection graph to obtain social circles, which can improve the accuracy of community discovery, thereby improving the accuracy of the obtained social circles and improving user experience.
[0019] Optionally, community discovery can be performed on the decentralized character connection graph based on the Louvain algorithm to obtain at least one social circle. In this way, the time complexity of the calculation is low, the community division result is stable, and the generated social circle result is stable and accurate. Of course, in some embodiments, other algorithms can also be used to discover communities and obtain social circles. For example, community discovery can be performed based on an information resource map (infomap) algorithm. The embodiments of the present application do not impose any restrictions on the specific method of community discovery.
[0020] In a possible implementation, the character image heterogeneous map also includes shooting times and shooting locations of multiple images. According to the character image heterogeneous map, identifying the central character includes: according to the character image heterogeneous map, respectively performing divergence calculation on the images corresponding to each of the multiple characters to obtain the time distribution divergence and location distribution divergence of the images corresponding to each of the multiple characters; determining the character whose time distribution divergence and location distribution divergence of the corresponding image meet the first preset condition as the central character; the divergence calculation includes: according to the shooting time of the image corresponding to the first character, counting the distribution of the image corresponding to the first character in multiple preset time intervals to obtain the first time distribution; determining the divergence of the first time distribution and the uniform time distribution to obtain the time distribution divergence of the image corresponding to the first character; according to the shooting location of the image corresponding to the first character, counting the distribution of the shooting location of the image corresponding to the first character in multiple known locations to obtain the first location distribution, and the multiple known locations include the shooting locations of all images in the gallery application; determining the divergence of the first location distribution and the uniform location distribution to obtain the location distribution divergence of the image corresponding to the first character.
[0021] The shooting time and shooting location are also the node information of the image nodes in the person image heterogeneous graph. Divergence calculation is also called divergence calculation operation.
[0022] In this implementation, the central person is determined by calculating the divergence of the image corresponding to each person. This process can accurately identify the central person without obtaining information such as the user's face unlocking, does not affect the security of the user's private information, and further improves the user experience.
[0023] In one possible implementation, based on the shooting time of the image corresponding to the first person, the distribution of the shooting time of the image corresponding to the first person in multiple preset time intervals is counted to obtain a first time distribution, including: counting the number of matching images in each preset time interval, wherein the matching image in the first preset time interval refers to an image corresponding to the first person, whose shooting time matches the first preset time interval, and the first preset time interval is any one of multiple preset time intervals; determining the first time distribution based on the number of matching images in each preset time interval.
[0024] In a possible implementation, multiple preset time intervals are obtained by dividing a day into multiple time intervals; counting the number of matching images in each preset time interval includes: if the timing information in the shooting time of the first target image belongs to the target preset time interval, and the second target image has not been counted, then the number of matching images in the target preset time interval is increased by one; the first target image is any one of the images corresponding to the first person, and the second target image is an image corresponding to the first person, the timing information belongs to the target preset time interval, and the date information is the same as the date information in the shooting time of the first target image.
[0025] In this implementation, when counting matching images, only one image with the same date information and the same preset time interval is counted. That is, for a preset time interval, if there are multiple matching images on the same day, only one image is counted. This is because the first time distribution is used to determine the time distribution divergence (i.e., uniformity) of the image corresponding to the first person, and whether the first person is the central person is determined by the time distribution divergence of the image corresponding to the first person. The more uniform the time distribution, the greater the possibility that the first person is the central person (e.g., the owner or the owner's family). In actual use, even if it is a non-central person, when using a camera to take pictures, it is possible to take pictures in a preset time interval on a certain day, resulting in multiple images. However, adding the concentrated pictures in a preset time interval to the time distribution statistics will result in uneven time distribution of the images corresponding to the person, which may cause the person to be mistakenly judged as the central person. Therefore, through the method provided by this implementation, only one image of the concentrated pictures is counted, and the images generated by the concentrated pictures are eliminated to prevent the accuracy of the time distribution statistics of the images corresponding to the first person from being affected, thereby making the determined central person more accurate.
[0026] In a possible implementation, the first time distribution is determined according to the number of matching images in each preset time interval, including: respectively determining the ratio of the number of matching images in each preset time interval to the total number of images corresponding to the first person to obtain the first time distribution.
[0027] In one possible implementation, based on the shooting location of the image corresponding to the first person, the distribution of the images corresponding to the first person in multiple known locations is counted to obtain the first location distribution, including: respectively determining the mark value corresponding to each known location, the mark value corresponding to the first known location is used to mark whether there is an image with the shooting location of the first known location in the image corresponding to the first person, and the first known location is any one of the multiple known locations; determining the first location distribution according to the mark value corresponding to each known location.
[0028] In this implementation, considering that the first location distribution is used to determine the location distribution divergence (i.e., uniformity) of the target co-occurrence image, whether the first person is the central person is determined by the location distribution divergence. The more uniform the location distribution, the greater the possibility that the first person is the central person. In actual use, even if it is a non-central person, when using a camera to take pictures, they may take pictures in a certain location and generate multiple images. However, adding concentrated pictures at a certain location to the location distribution statistics will make the location distribution of the images corresponding to the first person uneven, and the first person may be mistakenly judged as the central person. Therefore, in this implementation, by counting whether there are images corresponding to the first person corresponding to each known location, the location distribution of the images corresponding to the first person is analyzed, without considering the number of images taken at each known location, to prevent the images generated by concentrated pictures from affecting the accuracy of the first location distribution statistics, thereby making the determined central person more accurate.
[0029] In a possible implementation, the tag value includes a first value and a second value, the first value representing that there are images corresponding to the first person and the shooting location is the first known location, and the second value representing that there are no images corresponding to the first person and the shooting location is the first known location; the first location distribution represents the proportion corresponding to each known location; according to the tag value corresponding to each known location, the first location distribution is determined, including: setting the proportion corresponding to the known locations with the tag value of the first value to 1 / the total number of the first value, and setting the proportion corresponding to the known locations with the tag value of the second value to 0, to obtain the first location distribution.
[0030] Optionally, the first value may be 1, and the second value may be 0.
[0031] In one possible implementation, the first preset condition includes: the divergence sum is in the top Q1 of the first ranking; the divergence sum is the sum of the time distribution divergence and the location distribution divergence, and the first ranking is obtained by ranking multiple characters in ascending order of the divergence sum, and Q1 is a positive integer; or, the divergence ranking sum is in the top Q2 of the second ranking; the divergence ranking sum is the sum of the time divergence ranking and the location divergence ranking, and the second ranking result is obtained by ranking multiple characters in ascending order of the divergence ranking sum, and Q2 is a positive integer.
[0032] Optionally, the first preset condition may include: the divergence ranking is located in the last Q1 position of the fourth ranking, and the fourth ranking is obtained by ranking multiple characters in order of divergence and from large to small; or, the divergence ranking is located in the last Q2 position of the fifth ranking, and the fifth ranking is obtained by ranking multiple characters in order of divergence and from large to small.
[0033] The time distribution divergence and location distribution divergence can reflect the uniformity of the distribution of images corresponding to the person. The more uniform the distribution of images corresponding to the person, the greater the probability that the person is the central person. Therefore, in this implementation, the central person can be selected simply and quickly through the divergence sum or the divergence ranking sum.
[0034] In a possible implementation, a first social circle is identified based on the intimacy between every two characters and a heterogeneous graph of character images, including: identifying a first social circle and at least one second social circle based on the intimacy between every two characters and a heterogeneous graph of character images, the second social circle being different from the first social circle; the method also includes: determining the circle intimacy of the first social circle and each second social circle, the circle intimacy being the average of the intimacy between each character in the social circle and the central character; determining a social circle whose circle intimacy meets a second preset condition in the first social circle and each second social circle as an intimate social circle; generating a second album based on the intimate social circle, the second album including images within the second circle, the images within the second circle being images including at least one character in the intimate social circle.
[0035] In this implementation, photo albums are generated based on close social circles, which further fits the user's actual social situation, thereby further improving the user experience.
[0036] In one possible implementation, the second preset condition includes: the circle intimacy is greater than a preset threshold, or the circle intimacy ranking is in the top X of the third ranking, and the third ranking is obtained by ranking the first social circle and each second social circle in order of circle intimacy from large to small, and X is a positive integer.
[0037] In this implementation, the social circle with greater intimacy is regarded as the close social circle, and the social circle with the closest social connection to the user is simply and quickly selected to generate a second album, thereby improving the user experience.
[0038] In a possible implementation, the intimacy between every two characters among a plurality of characters is determined according to a character image heterogeneous graph, including: determining the intimacy between characters corresponding to non-co-occurring images and other characters among the plurality of characters as 0, wherein the non-co-occurring images are images including one character; determining a co-occurring image set from a plurality of images according to the character image heterogeneous graph, wherein the co-occurring image set includes a plurality of co-occurring images, wherein the co-occurring images are images including at least two characters; performing intimacy calculations respectively according to the co-occurring images of every two characters in the co-occurring image set to obtain the intimacy between every two characters included in the images in the co-occurring image set; the intimacy calculation includes: determining the mutual information between the second character and the third character according to the number of images corresponding to the second character in the co-occurring image set, the number of images corresponding to the third character, the number of target co-occurring images, and the total number of images in the co-occurring image set; the second character and the third character are any two characters included in the images in the co-occurring image set, and the target co-occurring image is a co-occurring image including the second character and the third character; determining the intimacy between the second character and the third character according to the mutual information.
[0039] Intimacy calculation is also called intimacy calculation operation.
[0040] Optionally, the intimacy between the first person and the second person may be directly represented by mutual information, that is, the mutual information between the first person and the second person is directly determined as the intimacy between the two.
[0041] Optionally, the intimacy between the second person and the third person may be determined based on the mutual information in combination with other parameters.
[0042] Mutual information can characterize the correlation between two things. Therefore, in this implementation, the correlation between the second person and the third person, that is, the intimacy, can be determined through the mutual information between the second person and the third person. In addition, in this implementation, the mutual information between the second person and the third person can be determined according to the number of images corresponding to the second person in the co-occurrence image set, the number of images corresponding to the third person, the number of target co-occurrence images, and the total number of images in the co-occurrence image set, and then the intimacy can be determined. In other words, the intimacy between the second person and the third person can be determined offline based on the number of images, etc., without relying on the network, so this method is more applicable.
[0043] In a possible implementation, a heterogeneous graph of character images includes shooting times and shooting locations of multiple images; based on mutual information, determining the intimacy between the second character and the third character, including: based on the shooting time of the target co-occurrence image, counting the distribution of the target co-occurrence image in multiple preset time intervals to obtain a second time distribution; determining the divergence between the second time distribution and the uniform time distribution to obtain the time distribution divergence of the target co-occurrence image; based on the shooting location of the target co-occurrence image, counting the distribution of the target co-occurrence image in multiple known locations to obtain a second location distribution, wherein the multiple known locations include the shooting locations of all images in the gallery; determining the divergence between the second location distribution and the uniform location distribution to obtain the location distribution divergence of the target co-occurrence image; determining the intimacy between the second character and the third character based on the mutual information, the time distribution divergence of the target co-occurrence image, and the location distribution divergence of the target co-occurrence image.
[0044] The second person may be person a in the specific implementation manner, and the third person may be person b in the specific implementation manner. The target co-occurrence image is also referred to as the person ab co-occurrence image in the specific implementation manner. The second time distribution is also referred to as the co-occurrence time distribution of person ab in the specific implementation manner, and the second location distribution is also referred to as the co-occurrence location distribution of person ab in the specific implementation manner.
[0045] In this implementation, when determining the intimacy between the second person and the third person, on the basis of mutual information, the time distribution divergence and location distribution divergence of the target co-occurrence image are further added. The time distribution divergence of the target co-occurrence image can characterize the uniformity of the distribution of the co-occurrence images of the second person and the third person in time. The location distribution divergence of the target co-occurrence image can characterize the uniformity of the distribution of the co-occurrence images of the second person and the third person in location. The closer the social relationship between the second person and the third person, the more uniform the distribution of their co-occurrence images in time and location. Therefore, in this implementation, the time distribution divergence and location distribution divergence of the target co-occurrence image are integrated into the calculation of intimacy, which improves the accuracy of intimacy calculation, thereby improving the accuracy of social circle recognition.
[0046] In one possible implementation, the intimacy between the second person and the third person is determined based on the mutual information, the time distribution divergence of the target co-occurrence image, and the location distribution divergence of the target co-occurrence image, including: determining the difference between 1 and the time distribution divergence of the target co-occurrence image to obtain a first difference; determining the difference between 1 and the location distribution divergence of the target co-occurrence image to obtain a second difference; determining the intimacy between the second person and the third person based on the product of the mutual information, the first difference, and the second difference.
[0047] Optionally, the intimacy between the second character and the third character may be determined according to the following formula:
[0048] closure(a,b)=PMI(a,b)×(1-D ab,t )×(1-D ab,l );
[0049] Among them, closure(a,b) represents the intimacy between the second person and the third person, PMI(a,b) represents the mutual information between the second person and the third person, and D ab,t represents the temporal distribution divergence of the target co-occurrence image, (1-D ab,t ) represents the first difference, D ab,t Represents the location distribution divergence of the target co-occurrence image, (1-D ab,t ) represents the second difference.
[0050] The values of the time distribution divergence and location distribution divergence of the target co-occurrence image are both greater than or equal to 0 and less than or equal to 1. Therefore, by calculating the first difference and the second difference, and based on the product of the mutual information, the first difference and the second difference, when determining the intimacy between the first person and the second person, the smaller the time distribution divergence and location distribution divergence of the target co-occurrence image, the greater the intimacy between the first person and the second, and vice versa. The method provided by this implementation can simply and accurately determine the intimacy between the second person and the third person.
[0051] In one possible implementation, the character image heterogeneous graph includes shooting time information and shooting locations of multiple images. The method also includes: determining the type of the first social circle based on the character image heterogeneous graph, the type of the first social circle characterizes the type of social relationship between the characters in the first social circle and the central character, the time distribution divergence and location distribution divergence of the image corresponding to the central character meet the first preset condition, the time distribution divergence is the distribution divergence of the shooting time of the image, and the location distribution divergence is the distribution divergence of the shooting location of the image; determining the theme of the first album based on the type of the first social circle.
[0052] Optionally, the types of social circles may include but are not limited to family members, colleagues, and classmates.
[0053] The method in this implementation can identify the type of social circle and generate the theme of the album according to the type of social circle. In this way, the theme of the generated album is related to the type of social relationship, so the generated album theme will be more in line with the user's actual social situation, further improving the user experience. Moreover, when identifying the type of social circle, the type of social circle is determined based on the recognition result of the social circle. Compared with obtaining different social circles and social circle types by simply classifying the relationships between characters, the method provided by this application has higher accuracy in identifying social circles and social circle types, and better user experience.
[0054] In one possible implementation, the type of the first social circle is determined based on a heterogeneous graph of character images, including: inputting the heterogeneous graph of character images into a graph neural network model to extract a character feature vector of each of a plurality of characters and an image feature vector of each of the plurality of images; inputting the character feature vector and the image feature vector into a multi-layer perceptron model to predict the type of social relationship between each of the plurality of characters and a central character; and determining the type of the first social circle based on the type of social relationship between the characters in the first social circle and the central character.
[0055] Optionally, the graph neural network module can be a modeling relational data with graph convolutional networks (R-GCN) model.
[0056] In this implementation method, based on the pre-trained graph neural network model and multi-layer perceptron model, it is possible to simply, quickly and accurately predict the type of social relationship between a character and a central character, and then determine the type of social circle based on the type of social relationship, so as to facilitate the subsequent generation of album themes based on the social circle type, so that the generated album themes are no longer restricted and are more in line with the user's actual social situation, further improving the user experience.
[0057] In one possible implementation, the type of the first social circle is determined based on the social relationship types between the characters in the first social circle and the central character, including: determining the number of characters corresponding to each type of social relationship in the first social circle; and determining the social relationship type with the largest number of corresponding characters as the type of the first social circle.
[0058] In one possible implementation, the method also includes: modifying the social relationship type of the person to be corrected in the first social circle to the type of the first social circle; the person to be corrected is a person in the first social circle, whose social relationship type with the central person is different from the type of the first social circle.
[0059] In this embodiment, the social relationship type of the person to be corrected is modified to the type of the first social circle. In this way, the relationship type of the person with deviation in judgment is corrected, so that the relationship between the person in the determined first social circle and the central person is consistent and accurate, which facilitates the subsequent generation of photo albums according to the social circle and the generation of the theme of the photo album according to the social circle type.
[0060] In a possible implementation, a character image heterogeneous graph is determined based on multiple images in a gallery application, including: generating face clustering results based on the multiple images in the gallery application, the face clustering results characterizing the correspondence between the multiple images, the multiple faces, and the multiple characters, wherein, among the multiple images, an image including a first face corresponds to the first face, and among the multiple characters, a character to which the first face belongs corresponds to the first face, and the first face is any one of the multiple faces; based on the face clustering results, determining the character image heterogeneous graph.
[0061] The face clustering results are presented in the form of a list or the like. The face clustering results represent the correspondence between multiple images, the faces included in each image, and the person to which each face belongs. Therefore, the person to which the faces included in each image belong, and the person included in each image can be obtained according to the face clustering results. For example, image A includes face a and face b, face a belongs to person a, and face b belongs to person b, that is, image A includes person a and person b.
[0062] Specifically, each person in the face clustering result can be regarded as a person node, each image in the face clustering result can be regarded as an image node, and the image node containing a certain person can be connected with the person node corresponding to the person through an edge to obtain a person image heterogeneous graph. In other words, the person image heterogeneous graph includes multiple person nodes, multiple image nodes, and multiple edges connecting the person nodes and the image nodes. The multiple person nodes correspond to the persons in the face clustering result one by one, and the multiple image nodes correspond to the multiple images in the face clustering result one by one. There is an affiliation relationship (or called a containment relationship) between the corresponding images and persons between an image node and a person node connected by each edge.
[0063] In this implementation, face clustering results are generated through multiple images, and based on the face clustering results, a heterogeneous graph of human images can be accurately and quickly determined, thereby improving the algorithm operation efficiency.
[0064] In a second aspect, the present application provides a device, which is included in an electronic device, and the device has the function of implementing the electronic device behavior in the above-mentioned first aspect and the possible implementation of the above-mentioned first aspect. The function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a receiving module or unit, a processing module or unit, etc.
[0065] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.
[0066] In a fourth aspect, the present application provides a chip, including a processor. The processor is used to read and execute a computer program stored in a memory to perform the method in the first aspect and any possible implementation thereof.
[0067] Optionally, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.
[0068] Further optionally, the chip also includes a communication interface.
[0069] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.
[0070] In a sixth aspect, the present application provides a computer program product, the computer program product comprising: a computer program code, when the computer program code runs on an electronic device, the electronic device executes any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 This is a schematic diagram of an application scenario of a photo album generation method provided in an embodiment of the present application;
[0072] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0073] Figure 3 is a software structure block diagram of an electronic device provided in an embodiment of the present application;
[0074] Figure 4 It is a flowchart of a method for generating a photo album provided in an embodiment of the present application;
[0075] Figure 5 This is a schematic diagram of an album interface provided in an embodiment of the present application;
[0076] Figure 6 is a flowchart of another example of an album generation algorithm provided in an embodiment of the present application;
[0077] Figure 7 It is a logic diagram of an example of a photo album generation algorithm provided in an embodiment of the present application;
[0078] Figure 8 This is an example of a person-image isomorphic graph provided in an embodiment of the present application;
[0079] Fig. 9 This is an example of a character connection diagram provided in an embodiment of the present application;
[0080] Fig.10This is an example of a decentralized character connection diagram provided in an embodiment of the present application;
[0081] Fig.11 This is a schematic diagram of a scenario for removing marginal character nodes provided in an embodiment of the present application;
[0082] Fig.12 This is an example of a person-social circle diagram provided in an embodiment of the present application;
[0083] Fig.13 This is a schematic diagram of an example of social circle and social circle type recognition result provided in an embodiment of the present application;
[0084] Fig.14 This is a co-occurrence time distribution diagram of person ab provided in an example of the present application;
[0085] Fig.15 1 is a schematic diagram comparing the co-occurrence time distribution and uniform time distribution of characters ab provided in an example of the present application;
[0086] Fig.16 This is an example of a co-occurrence location distribution map provided in an embodiment of the present application;
[0087] Fig.17 It is a comparative schematic diagram of the co-occurrence location distribution and uniform location distribution of an example of characters ab provided in an embodiment of the present application. DETAILED DESCRIPTION
[0088] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0089] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.
[0090] References to "one embodiment" or "some embodiments" etc. described in the specification of the present application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in the specification of the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0091] In order to better understand the embodiments of the present application, the terms or concepts that may be involved in the embodiments are explained below.
[0092] 1. Heterogeneous Graph
[0093] A heterogeneous graph refers to a graph in which the sum of the node type and the edge type is greater than 2. That is, a graph in which the node type + the edge type is greater than 2. It can be understood that a heterogeneous graph belongs to graph data, and its expression form can be an image, a list, a vector, a class, etc., and the embodiments of the present application do not impose any restrictions on this. For details, please refer to the relevant examples in the subsequent embodiments.
[0094] 2. Isomorphic graph
[0095] An isomorphic graph refers to a graph with only one node type and one edge type. It can also be understood that an isomorphic graph also belongs to graph data, and its expression form can be an image, a list, a list, a vector, a class, etc., and the embodiments of the present application do not make any limitation on this.
[0096] 3. Pointwise mutual information (PMI)
[0097] Mutual information is used to measure the correlation between two things. The larger the mutual information, the greater the correlation between the two things. The smaller the mutual information, the smaller the correlation between the two things.
[0098] 4. Shooting time information
[0099] The shooting time information of an image refers to information related to the shooting time of the image. Optionally, the shooting time information may include the shooting time, the week information corresponding to the shooting time (also called week information), etc. The shooting time may include year information, month information, day information, hour information, minute information, and second information, etc. It can be understood that the type of information included in the shooting time may be different depending on the timing accuracy.
[0100] For ease of understanding, in the embodiments of the present application, the year information, month information and day information are collectively referred to as date information, and the hour information, minute information and second information are collectively referred to as timing information. For example, the shooting time of image A is 17:18:12 on April 12, 2023, then the year information contained in the shooting time is 2023, the month information is April, the day information is the 12th, the hour information is 17 hours, the minute information is 18 minutes, and the second information is 12 seconds. Among them, the date information is April 12, 2023, and the timing information is 17 hours, 18 minutes and 12 seconds. It can also be understood that the types of information included in the date information and the timing information may be different due to different timing accuracy.
[0101] 5. Temporal distribution of images
[0102] The embodiments of the present application involve the time distribution of images. The time distribution of images represents the distribution of the shooting moments of the images, that is, the distribution of the number of images in different time intervals is determined according to the shooting moments. For example, according to the timing information in the shooting moments, the number of images in the time interval TA is determined to be pic1, and the number of images in the time interval TA2 is pic2.
[0103] 6. Location distribution
[0104] The embodiments of the present application involve the location distribution of images. The location distribution of images represents the distribution of the shooting locations of the images, that is, the distribution of the number of images of different locations determined according to the shooting locations. For example, it is determined that the number of images shot at location L1 is pic3, and the number of images shot at location L2 is pic4.
[0105] 7. Divergence
[0106] Divergence is a data distribution characteristic and a commonly used indicator in statistics to describe group structure. It is used to indicate the degree of deviation of data from the mean. Divergence can characterize the uniformity of data distribution.
[0107] The time distribution divergence refers to the degree of deviation of the time distribution from the mean of the time distribution (i.e., uniform time distribution). The time distribution divergence can reflect the uniformity of the time distribution. The smaller the time distribution divergence, the more uniform the time distribution is. The larger the time distribution divergence, the more uneven the time distribution is, i.e., the more concentrated it is.
[0108] The location distribution divergence refers to the degree of deviation of the location distribution from the mean of the location distribution (i.e., uniform location distribution). The location distribution divergence can reflect the uniformity of the location distribution. The smaller the location distribution divergence, the more uniform the location distribution is. The larger the location distribution divergence, the more uneven the location distribution is, i.e., the more concentrated it is.
[0109] 8. KL Divergence
[0110] KL is the abbreviation of Kullback-Leibler. KL divergence is used to measure the difference between two distributions. The larger the KL divergence, the greater the difference between the two distributions. The smaller the KL divergence, the smaller the difference between the two distributions. The value of KL divergence can be [0,1]. When the two distributions are consistent, the KL divergence is 0.
[0111] 9. Co-occurrence relationship
[0112] Co-occurrence refers to co-appearance. In the embodiment of the present application, the co-occurrence relationship refers to the co-occurrence relationship between characters in an image. In other words, the co-appearance of characters in an image is called the co-occurrence relationship between characters.
[0113] Before describing in detail the photo album generation method provided in the embodiment of the present application, the application scenario of the method and the structure of the applicable electronic device are first described.
[0114] Optionally, the album generation method provided in the embodiment of the present application can be applied to a gallery APP of an electronic device to implement functions such as "moments" or "wonderful moments" or "memories" of the gallery APP. The following is an explanation in conjunction with the interface diagram.
[0115] For example, Figure 1 The following is a schematic diagram of an application scenario of a method for generating an album provided in an embodiment of the present application. Taking a mobile phone as an example, Figure 1 As shown in FIG. 1 (a), the main interface of the mobile phone includes an icon 101 of the gallery APP. The user clicks the icon 101, and in response to the user's click, the mobile phone opens the gallery APP and displays Figure 1 The interface shown in (b) in FIG. 1 includes a time control 102. The user clicks the time control 102, and in response to the user's click, the mobile phone displays the interface of the "time" function, such as Figure 1 As shown in Figure (c) in the figure, the interface can include albums with multiple themes such as "Weekend Time", "Selected Portraits", and "Happy New Year".
[0116] about Figure 1In the interface shown in Figure (c), the albums with different themes are mainly generated according to the shooting time of the image (i.e., photo) and the frequency of the appearance of the characters in the related art. For example, the album with the theme of "Weekend Time" in the figure is generated by selecting images shot on weekends from the gallery based on the shooting time of the images. The album with the theme of "Selected Portraits" is generated by identifying the faces in each image in the gallery, selecting images with faces that appear more frequently, and using these images. The album with the theme of "Happy New Year" is generated by selecting images shot within a preset time period including the Spring Festival from the gallery based on the shooting time of the images. However, the album themes generated by the above method are limited, and the user experience needs to be improved.
[0117] In view of this, an embodiment of the present application provides a method for generating an album, which can identify the social circle of the person in the image based on the image in the gallery, and determine the type of the social circle, so that the album can be generated according to the social circle, and the theme of the album can be determined according to the type of the social circle. In this way, the generated album is related to the social circle of the person in the image, and the theme of the album is related to the social relationship of the person, so the generated album and the theme of the album are more flexible and extensive, which solves the problem of limited album themes and improves the user experience. Moreover, the album and the theme of the album generated in this way will be more in line with the user's actual social situation, further improving the user experience.
[0118] The album generation method provided in the embodiments of the present application can be applied to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), and other electronic devices that can install application programs (APP). The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.
[0119] For example, Figure 21 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0120] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0121] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0122] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0123] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0124] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0125] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0126] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.
[0127] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0128] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0129] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Figure 2 The structures of antenna 1 and antenna 2 in the electronic device 100 are only examples. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0130] The mobile communication module 150 may provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.
[0131] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0132] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.
[0133] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0134] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0135] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0136] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0137] Figure 3 1 is a software structure diagram of the electronic device 100 of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.
[0138] The application layer can include a series of application packages. Figure 3 As shown, the application package may include a gallery APP. In addition, the application layer may also include applications such as camera, call, map, Bluetooth, short message, etc., which are not limited in any way in the embodiments of the present application.
[0139] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0140] like Figure 3As shown, in the embodiment of the present application, the gallery APP of the application layer may include a face clustering module, an album generation module, and a gallery data storage module. The application framework layer may include modules such as a face clustering algorithm and a social circle recognition algorithm. The functional module of the gallery APP implements the relevant functions of the gallery APP by calling the algorithm of the application framework layer.
[0141] Specifically, the gallery data storage module (also referred to as gallery) is used to store relevant data of the gallery APP, such as images, face clustering results, person-image heterogeneous graphs (or called person-image heterogeneous graphs), person connection graphs, social circle data, etc.
[0142] The face clustering module recognizes faces in the image by calling the face clustering algorithm of the application framework layer, and clusters the recognized faces to obtain face clustering results. Specifically, clustering faces means clustering faces with similar features into a class, so that at least one class is obtained, and each class can correspond to a person. In other words, the face clustering result represents the person corresponding to the faces with similar features in the image.
[0143] The album generation module identifies the social circle and type of the social circle of the character by calling the social circle recognition algorithm of the application framework layer, generates an album based on the social circle, and determines the theme of the album based on the type of the social circle. Specifically, the social circle recognition algorithm can generate a character-image heterogeneous graph based on the clustering results of the face clustering module, and generate a character connection graph based on the character-image heterogeneous graph. At the same time, the social circle recognition algorithm identifies the central character based on the character-image heterogeneous graph. The social circle of the character in the image is discovered based on the central character and the character connection graph. Afterwards, the type of social circle is determined based on the social circle, the central character and the character-image heterogeneous graph. The social circle recognition algorithm returns the identified social circle and the type of social circle to the album generation module. The album generation module generates an album based on the social circle, and generates the theme of the album based on the type of the social circle. The detailed implementation process of the above process will be further described in the subsequent embodiments.
[0144] It should be noted that the structure of the above-mentioned gallery APP, as well as the layers where the social circle recognition algorithm and the face clustering algorithm are located, are all examples and are not intended to be limiting. In actual applications, the structure of the gallery APP may include more or fewer modules than the above-mentioned, and the social circle recognition algorithm and the face clustering algorithm may also be located in other layers of the software architecture.
[0145] In addition to the face clustering algorithm and the social circle recognition algorithm, the application framework layer may also include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0146] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0147] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0148] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0149] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).
[0150] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0151] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0152] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0153] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0154] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0155] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0156] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0157] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0158] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0159] A 2D graphics engine is a drawing engine for 2D drawings.
[0160] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0161] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 2 and Figure 3 Taking the electronic device with the structure shown as an example, the album generation method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.
[0162] Figure 4 : is a flowchart of a method for generating an album provided in an embodiment of the present application, the method comprising:
[0163] S101. The face clustering module of the gallery APP calls the face clustering algorithm of the application framework layer to process the images in the gallery data storage module.
[0164] Optionally, the face clustering module can also call the face clustering algorithm to process the image when the electronic device is in the off state, so as to reduce the impact on the normal use of the user and improve the user experience.
[0165] S102, the face clustering algorithm responds to the call of the face clustering module, performs face recognition and clustering on the images in the gallery data storage module, and obtains a face clustering result, wherein the face clustering result includes at least one class, and each class corresponds to a person.
[0166] Specifically, the face clustering algorithm performs face recognition on each image to obtain the faces that appear in the image. Optionally, the face clustering algorithm can identify the faces recognized in each image separately to distinguish multiple faces in the same image. For example, the faces recognized in each image can be numbered, and an identity document (ID) corresponding to each face can be set. In other words, for a certain image, each face included therein corresponds to a face ID (also called face_id). It should be noted that the face IDs corresponding to faces in different images can be the same or different. For example, the IDs of the faces that appear in image A, image B, and image C can be set to 0, 1, 2, 3, 4... in sequence, or the face IDs can be set separately for the faces that appear in each image. For example, the IDs of the faces appearing in image A are sequentially set to 0, 1, 2, 3, ..., the IDs of the faces appearing in image B are sequentially set to 0, 1, 2, 3, ..., the IDs of the faces appearing in image C are sequentially set to 0, 1, 2, 3, .... When the face IDs in different images are the same, the faces can be distinguished by the identifier of the image to which they belong. The identifier of the image can be, for example, the hash ID of the image.
[0167] Afterwards, the face clustering algorithm clusters the faces, clusters faces with similar features into the same class, and obtains multiple classes. Specifically, the face clustering algorithm can cluster the faces recognized from the image with historical faces to obtain multiple classes. Among them, historical faces refer to faces recognized by the face clustering algorithm within a historical period of time. In the multiple classes obtained by clustering, each class corresponds to a person. Optionally, the face clustering algorithm can identify each person, for example, the characters can be numbered and the ID corresponding to each person can be set. In other words, each person can be represented by a character ID. The character ID is also called the target ID (i.e., tag_id). It should be noted that when the face clustering algorithm performs face clustering, it may be impossible to cluster, that is, for a certain face, there are no other faces with similar features. In this case, the face clustering algorithm can set the character ID corresponding to the face to a preset identifier, for example, to -1. In other words, a character ID of -1 indicates that there is no corresponding class. During the subsequent algorithm operation, the characters whose IDs are preset identifiers can be excluded from the calculation scope, and only the characters whose IDs are not the preset identifiers can be considered. This can improve the accuracy of social circle and social circle type recognition and improve user experience.
[0168] Illustratively, Table 1 is an example of face clustering results provided in an embodiment of the present application.
[0169] Table 1
[0170] Serial number Image hash ID Face ID Character ID 1 42d8166356c2d1ff9875f602e97ea293 0 -1 2 42d8166356c2d1ff9875f602e97ea293 1 ser_246762133257613 3 b4a0d50b46f4f8ff86a25916c2cedecc 0 ser_112952351578583 4 2b869431a9182637d2a12c8444e352ec 0 -1 5 2586222e6dac13d5630edac4a8662208 0 ser_111346622209036 6 b320cc0aca977684063b4e3069f6fa08 0 -1 7 c65dbc7358974489502964155ad4cac0 0 ser_275636865923451 8 c65dbc7358974489502964155ad4cac0 1 ser_246762133257613 9 26fe24ebd66f90b540cc79637efdc46d 0 ser_136254524879543 10 26fe24ebd66f90b540cc79637efdc46d 1 ser_246762133257616 11 26fe24ebd66f90b540cc79637efdc46d 2 ser_112952351578583 12 26fe24ebd66f90b540cc79637efdc46d 3 ser_275636865923451
[0171] As shown in Table 1, the image hash ID is the identifier of the image, and each image hash ID corresponds to an image. The face ID represents the ID of the face that appears in the image. Taking the clustering results of sequence numbers 9, 10, 11, and 12 in Table 1 as an example, it can be seen that the image with the image hash ID "26fe24ebd66f90b540cc79637efdc46d" includes 4 faces, and the face IDs corresponding to the 4 faces are 0, 1, 2, and 3, respectively. The 4 faces are clustered to obtain the classes to which the 4 faces belong, that is, the character IDs corresponding to the 4 face IDs are obtained, which are "ser_136254524879543", "ser_246762133257616", "ser_112952351578583", and "ser_275636865923451", respectively.
[0172] It should be noted that Table 1 is only an example of face clustering results and does not constitute any limitation to the present application. In practical applications, the face clustering results obtained by the face clustering algorithm may include more or less content than Table 1, or may be in other forms of expression.
[0173] In addition, it is understood that the face clustering algorithm or other related modules can also analyze the gender and age of the person based on the recognized face information. The information such as the gender and age of the person can be saved in the gallery data storage module, or can be used for subsequent data flow together with the face clustering results, and the embodiment of the present application does not impose any limitation on this.
[0174] S103: The face clustering algorithm returns the face clustering result to the face clustering module.
[0175] S104: The face clustering module sends the face clustering result to the album generation module.
[0176] It is understood that in some embodiments, when the image in the gallery data storage module is updated (for example, a new image is added), the face clustering module can be triggered to update the face clustering result. Moreover, the face clustering module can send the updated result to the album module.
[0177] S105. The album generation module calls a social circle recognition algorithm to identify the person's social circle and the type of social circle.
[0178] Specifically, the social circle of a character refers to a collection of characters with social relationships. The type of social circle is used to characterize the relationship between the characters in the social circle and the central character. The central character refers to a character in the character heterogeneous graph whose corresponding image's time distribution divergence and location distribution divergence meet the first preset condition.
[0179] Optionally, the types of social circles may include family, colleagues, classmates, friends, etc.
[0180] Optionally, the album generation module may execute the operation of calling the social circle recognition algorithm in response to the instruction input by the user. Optionally, the album generation module may also execute the operation of calling the social circle recognition algorithm according to a preset period. Optionally, the album generation module may also execute the operation of calling the social circle recognition algorithm when receiving the face clustering update result sent by the face clustering module, and the embodiment of the present application does not impose any limitation on this.
[0181] S106. The social circle recognition algorithm responds to the call of the album generation module and identifies the person's social circle and the type of social circle based on the face clustering result.
[0182] In one embodiment, when calling the social circle recognition algorithm, the album generation module can carry relevant information of the face clustering results, so that the social circle device algorithm can identify the person's social circle and the type of social circle based on the face clustering results carried when calling.
[0183] In another embodiment, the face clustering results may also be stored in the gallery data storage module through the album generation module or the face clustering module, and the social circle recognition algorithm obtains the face clustering results from the gallery data storage module to identify the person's social circle and the type of social circle.
[0184] The social circle identified by the social circle identification algorithm may be one or more social circles. For example, the results identified by the social circle identification algorithm may include social circle 1 and social circle 2. The type of social circle 1 is family, and the type of social circle 2 is colleague. In other words, the relationship between the characters in social circle 1 and the central character is a family relationship, and the relationship between the characters in social circle 2 and the central character is a colleague relationship.
[0185] The specific implementation of this step refers to the subsequent embodiments.
[0186] S107. The social circle identification algorithm returns the social circle and the type of social circle to the album generation module.
[0187] S108. The photo album generation module generates a photo album according to the social circle, and generates a theme of the photo album according to the type of the social circle.
[0188] Optionally, the album generation module can select images including the target person in the gallery data storage module to generate an album based on the person included in the social circle (referred to as the target person), and generate an album with a theme based on the type of the social circle. For example, select images including the person in social circle 1 to generate album 1. The type of social circle 1 is family, so the theme of album 1 is set to "family". Exemplarily, the interface of the album containing "family" can be as follows: Figure 5 shown.
[0189] Optionally, when generating an album based on a social circle, it can be further combined with the image shooting time information, the image shooting location, etc., and when generating the theme of the album, it can also be combined with the image shooting time information, the image shooting location. For example, the images in social circle 1 that were shot in the preset Spring Festival time period are used to generate an album with the theme of "Memories of Spring Festival and Family".
[0190] Optionally, in some embodiments, after generating the photo album, the photo album generation module may also generate other data based on the images in the photo album, for example, generate a video file, etc. This application does not impose any limitation on this.
[0191] In some embodiments, the album generation module can also classify the images containing people in the gallery data storage module according to the person according to the face clustering result, and divide the images containing the same person into a group. The user can manually mark the person's name and the relationship between the person and the owner of the device.
[0192] In addition, after identifying the social circle and the type of social circle, the album generation module can also group the images containing people in the gallery data storage module according to the social circle, and group the images containing people in the same social circle into one group. The user can manually mark the social circle type and other information. In this way, the subsequent album generation module can generate an album based on the information marked by the user to further optimize the effect of generating the album.
[0193] The album generation method provided in this embodiment identifies the social circle and the type of social circle of the person in the image based on the face clustering result, thereby generating an album based on the social circle, and generating the theme of the album according to the type of social circle. In this way, the theme of the generated album is related to the social circle of the person in the image, so the generated album and the theme of the album are more flexible and extensive, solving the problem of limited album themes and improving user experience. Moreover, the album and the theme of the album generated in this way will be more in line with the user's actual social situation, further improving user experience.
[0194] The specific process of the social circle identification algorithm identifying social circles and social circle types is described below with reference to the accompanying drawings.
[0195] For example, Figure 6 This is a flowchart of another example of an album generation algorithm provided in an embodiment of the present application. Figure 7 This is a logical diagram of an example of an album generation algorithm provided in the embodiment of this application. Please also refer to Figure 6 and Figure 7 In this embodiment, in the above step S106, “identifying the social circle and the type of social circle of the person based on the face clustering result” includes:
[0196] S201. Generate a person-image heterogeneous graph based on the face clustering result, where the person-image heterogeneous graph is used to represent the correspondence between a person and an image containing the person.
[0197] Specifically, each person in the face clustering result can be regarded as a person node, each image including a face can be regarded as an image node, and the image node containing a certain face can be connected with the person node corresponding to the face through an edge to obtain a person-image heterogeneous graph. For example, image A includes face 1. According to the face clustering result, the person corresponding to face 1 is a, so the node of image A is connected with the node of person a through an edge; the same is true for other images and other people in the face clustering result. In this way, a person-image heterogeneous graph can be obtained. In other words, the person-image heterogeneous graph contains person nodes and image nodes, and the edge between a person node and an image node indicates that the image contains the person, or the edge between a person node and an image node indicates that the person belongs to the image.
[0198] Optionally, the person-image heterogeneous graph may also include node information of person nodes and image nodes. Specifically, the information of the person node is also the information of the person corresponding to the person node, including but not limited to the person ID, the person gender and the person age. The information of the image node is also the information of the image corresponding to the image node, including but not limited to the image hash ID, the image shooting time information, the image shooting location and the camera used to shoot the image (for example, whether it was shot with the front camera).
[0199] For example, Figure 8 An example of a person-image isomorphic graph provided in an embodiment of the present application. Figure 8 In the figure, the icons with round borders represent person nodes, and the icons with square borders represent image nodes. It can be understood that Figure 8 It is only used as an example to illustrate the composition structure of the person-image heterogeneous graph, etc. It is not intended to be limiting and does not represent actual data.
[0200] It should be noted that Figure 8 It is a concrete representation of the character-image heterogeneous graph. In practical applications, the character-image heterogeneous graph can be expressed in various forms. The embodiment of the present application does not impose any limitation on the specific expression form of the character-image heterogeneous graph, as long as it can reflect its content.
[0201] In a specific embodiment, the person-image heterogeneous graph can also be represented by a list. Specifically, the table corresponding to the person-image heterogeneous graph may include a person node list, an image list, a connection matrix list, etc. The person node list is used to store information about each person node. The image list is used to store information about each image node. The connection matrix list is used to store the connection relationship between the person node and the image node, wherein the connection relationship can be represented in the form of a matrix. The connection matrix list reflects the information of the edges in the person-image heterogeneous graph, representing the person included in each image, or the image to which each person belongs.
[0202] Exemplarily, the character node list may be as shown in Table 2, the image node list may be as shown in Table 3, and the connection matrix list may be as shown in Table 4.
[0203] Table 2
[0204]
[0205] Table 3
[0206]
[0207] Table 4
[0208] Serial number Image Node The matrix of character nodes connected to the image nodes 1 Image Node A [character node a, character node b, character node c] 2 Image Node B [character node b, character node c] 3 Image Node C [Character node a] … … … n Image Node N [character node a, character node n]
[0209] It should be noted that the above Tables 2, 3 and 4 are only used as examples to illustrate the presentation form of the person-image heterogeneous graph and the content of the node information, etc., and do not represent actual data, nor do they constitute any limitation on this application. In actual use, Tables 1, 2 and 3 may include more or less content.
[0210] S202: Determine a co-occurring image set according to the person-image heterogeneous graph.
[0211] Among them, a co-occurrence image refers to an image in which there is a co-occurrence relationship between characters. The existence of a co-occurrence relationship between characters means that two or more characters appear together in an image. In other words, a co-occurrence image refers to an image that contains at least two characters. For example, if an image A includes characters a, b, and c, then image A is determined to be a co-occurrence image. If an image B only includes character a, then image B is determined not to be a co-occurrence image.
[0212] Specifically, according to the person-image heterogeneous graph, the number of person nodes that are connected to the image node can be determined. If a certain image node is connected to two or more person nodes, the image corresponding to the image node is a co-occurrence image. The determined multiple co-occurrence images are used to generate a co-occurrence image set. It can be understood that in other embodiments, the determined multiple co-occurrence images may not be expressed by combination, but by other forms (such as lists). The embodiments of the present application do not impose any restrictions on this.
[0213] In addition, for each two characters in a co-occurrence image, the co-occurrence image can be called the co-occurrence image of the two characters. For example, if image A includes characters a, b, and c, then image A is the co-occurrence image of characters a and b, the co-occurrence image of characters a and c, and the co-occurrence image of characters b and c.
[0214] For ease of description, in the following embodiments, the co-occurrence image of person a and person b is referred to as person ab co-occurrence image, the co-occurrence image of person a and person c is referred to as person ac co-occurrence image, the co-occurrence image of person b and person c is referred to as person bc co-occurrence image, and so on. The co-occurrence image of any two characters is also referred to as a target co-occurrence image.
[0215] According to the character-image heterogeneous graph, the co-occurrence image of every two characters can also be determined. Specifically, for any two characters (taking character a and character b as an example), the character node matrix can be determined according to the connection matrix list, including both the character node corresponding to character a and the character node corresponding to character b, and the image nodes corresponding to these matrices can be determined, so that the co-occurrence image of character a and character b can be determined. Optionally, the co-occurrence image of every two characters can be represented in the form of a matrix. Taking the connection matrix list shown in Table 4 as an example, according to the character node matrix, it is determined that the matrix with sequence number 1 includes character a and character b, and the image node corresponding to the matrix is image node A. Therefore, it is determined that the image A corresponding to image node A is the co-occurrence image of character ab. With reference to this method, all co-occurrence images of character ab can be obtained, and the co-occurrence images of any other two characters can be obtained.
[0216] S203: Determine the intimacy between two persons in the face clustering result according to the image information of each image in the co-occurrence image set.
[0217] Specifically, the intimacy between two characters is used to characterize the intimacy of the co-occurrence relationship between the two characters, that is, to characterize the degree of association between the two characters.
[0218] It can be understood that the intimacy between two characters is a value greater than or equal to 0. The greater the intimacy, the closer the two characters are. If the intimacy is 0, it means that there is no social relationship between the two characters.
[0219] Specifically, for the characters included in the images in the co-occurrence image set, the intimacy between these characters can be determined based on the image information of each image in the co-occurrence image set. For the characters included in the images that do not belong to the co-occurrence image set (i.e., non-co-occurrence images), there is no co-occurrence relationship between these characters and other characters, so the intimacy between these characters and other characters is determined to be 0. In this way, the intimacy between every two characters in all the characters in the face clustering result can be obtained.
[0220] As a possible implementation method, the PMI between every two characters in all the characters included in the images of the co-occurrence image set can be determined based on the shooting time information, shooting location and other information of each image in the co-occurrence image set. Then, the intimacy between the two characters is determined based on the PMI between the two characters. Among them, the image information of the co-occurrence image can be obtained through the node information corresponding to the image node in the character-image heterogeneous graph. Specifically, it can be obtained from the image node list shown in Table 3. Of course, the corresponding image information can also be directly searched from the gallery data storage module based on the hash ID of the co-occurrence image. The embodiments of the present application do not impose any restrictions on this.
[0221] The specific implementation process of step S203 will be further described in detail in subsequent embodiments.
[0222] S204, simplifying the character-image heterogeneous graph into a character connection graph according to the intimacy between two characters.
[0223] The character connection graph is used to represent the social relationship between characters and the corresponding intimacy. The character connection graph is a homogeneous graph.
[0224] Specifically, based on the person-graph heterogeneous graph, the image nodes are removed, and then two person nodes whose intimacy is not 0 are connected by an edge, and the intimacy is used as the weight of the edge. In this way, a person connection graph can be obtained.
[0225] In one embodiment, Figure 8 The character connection graph obtained by simplifying the character-image heterogeneous graph shown in the figure can be shown as Fig. 9 shown.
[0226] It is also understandable that Fig. 9It is a concrete representation of a character connection graph. In practical applications, the character connection graph can be expressed in a variety of forms. For example, the character connection graph can be expressed in a list. Specifically, based on the above Tables 2, 3, and 4, Table 3 can be deleted, the character gender, character age, and other information in Table 2 can be deleted to obtain a simplified character node list, and the image nodes in Table 4 and the character nodes that do not have a co-occurrence relationship with other characters can be deleted to obtain a simplified connection matrix list. The simplified character node list and connection matrix list are used as a character connection graph. The embodiment of the present application does not impose any limitation on the specific form of expression of the character connection graph, as long as it can reflect its content.
[0227] S205: Identify the central person according to the person-image heterogeneous graph.
[0228] The central person refers to the person who is at the social center among the people included in the face clustering results. Generally, the central person has more and closer social relationships with other people. It can be understood that the number of central people can be one or more, and the specific number can be set according to actual needs. The central person can be the owner of the electronic device or a non-owner, such as the owner's family.
[0229] The study found that images containing central figures may be generated at multiple times and multiple locations, so the images containing central figures are relatively evenly distributed in terms of time and location. In other words, the time distribution and location distribution divergence of images containing central figures are small. Compared with non-central figures, images containing central figures are generally not concentratedly generated at only one or several times, and / or, are not concentratedly generated at only one or several locations. For example, when the central figure is the owner of the machine, the owner may take photos with his mobile phone at any time to generate images containing the owner. For non-owners, such as the owner's friends, images containing the friend may be generated by taking photos with the owner's mobile phone only during a period of time when traveling with the owner. Therefore, the distribution of images containing the owner himself is more evenly distributed than that of images containing the owner's friends.
[0230] Based on this, in the present application, the central person can be determined according to the time distribution divergence and location distribution divergence of the images of each person in the person-image heterogeneous graph. Specifically, the person whose time distribution divergence and location distribution divergence of the corresponding image meet the first preset condition among the people included in the face clustering result can be determined as the central person. Among them, the image of a certain person is also referred to as the image corresponding to the person, which refers to the image containing the person. For example, the image of person a is also referred to as the image corresponding to person a, which refers to the image containing person a, the image of person b is also referred to as the image corresponding to person b, which refers to the image containing person b, and so on.
[0231] Optionally, the time distribution divergence of the images of each character may be represented by a time KL divergence, and the location distribution divergence of the images of each character may be represented by a location KL divergence.
[0232] The specific implementation process of step S205 will be further described in detail in subsequent embodiments.
[0233] S206, removing the central character node in the character connection graph to obtain a decentralized character connection graph.
[0234] Specifically, based on the character connection graph, the node corresponding to the identified central character is removed, and the edges connecting the central character node and other character nodes are removed to obtain a decentralized character connection graph.
[0235] In one embodiment, Fig. 9 Take the character connection diagram shown in the figure as an example. If the central character is Fig. 9 The character corresponding to the character node filled with solid black in the figure connection diagram is obtained by removing the central character node based on the character connection diagram. Fig.10 .
[0236] Similar to the person-image heterogeneous graph and the person connection graph, Fig.10 Connect only as decentralized characters Figure 1 In practical applications, there can be many forms of expression of the decentralized character connection graph, which is not limited in any way in the embodiments of the present application and will not be described in detail here.
[0237] S207: Perform community detection based on the decentralized character connection graph to obtain at least one social circle.
[0238] The result of community discovery is multiple communities, each of which corresponds to a social circle. Each social circle includes multiple (at least two) people.
[0239] In one embodiment, community discovery can be performed based on the Louvain algorithm to obtain at least one social circle. Specifically, the following steps can be included:
[0240] 1) Take the decentralized character connection graph as the target connection graph.
[0241] 2) Take each community in the target connection graph as a node, perform modularity optimization operations on each node in the target connection graph, and obtain an updated connection graph.
[0242] Specifically, the modularity optimization operation includes: for any node x, try to add node x to each adjacent community respectively, and calculate the corresponding modularity benefit; if the maximum modularity benefit among all modularity benefits is a negative value, then the node x will not be changed, that is, the node x will not be added to any adjacent community; if the maximum modularity benefit among the modularity benefits of the adjacent communities is not a negative value, then the node x will be added to the adjacent community corresponding to the maximum modularity benefit.
[0243] Among them, the neighboring community of node x refers to the community where the nodes adjacent to node x are located. It should be noted that for a single node (i.e. a node that has not joined a community), it itself is a community. In other words, the community where a single node is located only includes one node, which is the node itself.
[0244] After traversing all nodes in the target connection graph through the modularity optimization operation, an updated connection graph is obtained.
[0245] Specifically, when node x is added to any adjacent community (taking adjacent community C as an example), the corresponding modularity benefit calculation process can be as follows:
[0246] a. Calculate the modularity of the target connection graph when node x joins the adjacent community C and obtain the first modularity.
[0247] b. Calculate the modularity of the target connection graph when node x does not join the adjacent community C, and obtain the second modularity.
[0248] c. Calculate the difference between the first modularity and the second modularity to obtain the modularity benefit corresponding to the adjacent community C.
[0249] In step a and step b, the first modularity and the second modularity can be calculated with reference to formula (1):
[0250]
[0251] In formula (1), Q represents the modularity of the target connection graph. W represents the sum of the weights of all edges in the target connection graph. i and j are both node numbers in the target connection graph, i represents node i, and j represents node j. i Represents the sum of the weights of all edges connected to node i in the target connection graph. j A represents the sum of the weights of all edges connected to node j in the target connection graph. i,j represents the weight of the edge between node i and node j. i represents the sum of the weights of all edges pointing to node i. i ,C j ) is the indicator function, C i represents the community where node i is located, C jrepresents the community where node j is located. When nodes i and j belong to the same community (i.e., C i With C j Same), δ(C i ,C j ) is 1; when node i and node j do not belong to the same community (i.e., C i With C j When δ(C i ,C j ) has a value of 0.
[0252] 3) The updated connection graph is used as the target connection graph, and each community in the updated connection graph is used as a node, and the process returns to step 2) until the nodes included in all communities no longer change, and at least one community is obtained.
[0253] That is to say, when step 2) is executed for the first time, the target connection graph is a decentralized character connection graph, and each node in the decentralized character connection graph is a single node, that is, a node that has not joined the community, so each node is regarded as a community. When step 2) is executed for the second time and each time thereafter, the target connection graph is an updated connection graph, which includes single nodes and discovered communities. Therefore, the communities in the updated connection graph are regarded as nodes, and modularity optimization operations are performed separately. In the process of performing the modularity optimization operation, for a single node in the updated connection graph, the community in which it is located only includes itself. In this way, repeated iterations are achieved, the community to which the node belongs is changed, and the modularity of each community is optimized until the modularity benefits of each community no longer increase and the nodes contained in each community no longer change.
[0254] In this embodiment, community discovery is performed based on the Louvain algorithm, which has low time complexity during calculation and stable community division results, so that the generated social circle results are stable and accurate. Of course, in some embodiments, other algorithms can also be used to perform community discovery and obtain social circles. For example, community discovery can be performed based on an information resource map (infomap) algorithm. The embodiments of this application do not impose any restrictions on the specific method of community discovery.
[0255] 4) Determine at least one social circle based on at least one community.
[0256] In one embodiment, community discovery is performed based on the Louvain algorithm. After obtaining at least one community, the set of characters corresponding to all nodes in each community can be directly determined as a social circle, so that the social circles corresponding to each community can be obtained.
[0257] In another embodiment, after the community is discovered based on the Louvain algorithm, the obtained community may be filtered to obtain the final social circle. Specifically, the following may be included:
[0258] For any community, the sum of the weights of the edges connecting each character node in the community with other character nodes in the community is calculated respectively; the N character nodes with the smallest sum of weights are removed from the community to obtain a de-marginalized community; the set of characters corresponding to all nodes in the de-marginalized community is determined as a social circle.
[0259] Among them, the N person nodes with the smallest sum of weights can be called edge person nodes. That is, the edge person nodes in the community are eliminated to obtain a marginal community.
[0260] For example, according to the above steps 1) to 3), a certain community C is determined to contain character nodes A1, character nodes A2, character nodes A3...character node An. Calculate the sum of the weights of the edges connecting character node A1 with other character nodes in community C (character node A2, character node A3...character node An); then calculate the sum of the weights of the edges connecting character node A2 with other character nodes in community C (character node A1, character node A3...character node An)...and so on, calculate the sum of the weights of the edges connecting each character node with other character nodes. Remove the N character nodes with the smallest sum of weights from community C. For example, if N=1, then remove the one character node with the smallest sum of weights from community C to obtain an edge community C. Determine the set of characters corresponding to all nodes in edge community C as a social circle.
[0261] For example, Fig.11 A schematic diagram of a scenario for removing marginal character nodes provided in an embodiment of the present application. Fig.11 In the figure, black dots represent character nodes, and the lines between character nodes are edges. The thickness of the edge represents the weight of the edge. The thicker the edge, the greater the weight of the edge, and vice versa. Fig.11 As shown, after community discovery, community C is obtained. Community C includes character node E. After calculation, the weight of the edge between character node E and other nodes in community C is the smallest, N=1. Therefore, character node E is removed from community C to obtain de-margined community C. The set of characters corresponding to all nodes in de-margined community C is determined as a social circle.
[0262] Each community is processed according to the above process to obtain the social circle corresponding to each community.
[0263] As mentioned above, the weight of an edge is the intimacy between two characters. Therefore, the sum of the weights of the edges connecting a character node to other character nodes characterizes the number of connections between the character node and other character nodes in the community and the size of the weights of each edge. A smaller sum of weights indicates that the character node is less connected to other characters in the community, and / or that the weights of the edges between the node and other nodes are smaller, which further indicates that the character corresponding to the character node is less associated with other characters in the social circle, and / or has a lower degree of intimacy with other characters, that is, the character is a character with a lower degree of connectivity, and is considered to be a marginal character for the social circle. In this embodiment, marginal characters are filtered, and only characters with higher degrees of connectivity are included in the social circle, so that the identified social circle is more in line with the user's actual social situation, further improving the user experience.
[0264] Optionally, the social circle results obtained by community discovery can be represented in the form of a connection graph, that is, a social circle graph is generated. The social circle graph is used to represent the people included in each social circle, as well as the social relationships and corresponding intimacy between the people.
[0265] It can be understood that after the social circle is discovered in the community, the central character node can be added to the social circle graph according to the character connection graph to obtain a character-social circle graph.
[0266] In one embodiment, the person-social circle graph can be as follows: Fig.12 shown. Fig.12 In the figure, black dots represent character nodes, and the lines between character nodes are edges. The thickness of the edge represents the weight of the edge. The thicker the edge, the greater the weight of the edge, and vice versa. It can be seen that the central character node is connected to multiple character nodes, and some of the edges have larger weights. Fig.12 The social circles shown include three: social circle 1, social circle 2 and social circle 3.
[0267] It should be noted that the above community discovery process may not consider node information, but only be based on the weights of nodes and edges. Therefore, the character connection graph generated in step S204 may not include node information, but only include nodes and edges, as well as edge weights (i.e., intimacy), which can further simplify the structure of the character connection graph and improve the algorithm operation efficiency.
[0268] In addition, in this embodiment, community discovery is performed based on the decentralized character connection graph to obtain a social circle. This is because: Generally speaking, in the character connection graph, there are many edges connecting the central character node to other nodes, and the central character has a high degree of intimacy with other characters, so the weight of the edge connected to the central character node will be very high. If the central character node is included in the community discovery, it is easy to divide the central character node into a certain community in a certain round of iteration. In this way, in the next round of iteration, when the community where the central character node is located is used as a node, there are many edges with other nodes, and / or the sum of the weights of the edges pointing to the community where the central character node is located will be very high, so other points that do not belong to the community will be divided into the community, resulting in inaccurate social circles generated in the end. Therefore, in this embodiment, community discovery is performed based on the decentralized character connection graph to obtain a social circle, which can improve the accuracy of community discovery, thereby improving the accuracy of the obtained social circle and improving user experience.
[0269] S208: Determine the type of each social circle according to the central person and the person-image heterogeneous graph. The type of the social circle represents the type of relationship between the people in the social circle and the central person.
[0270] As a possible implementation method, the type of each social circle can be determined based on a graph neural network model and a multilayer perceptron (MLP) model. Specifically, the following process can be included:
[0271] 1) Input the person-image heterogeneous graph into the pre-trained graph neural network model to obtain the feature vector of each person node and image node.
[0272] The graph neural network model is used to extract the feature vector of the node. Optionally, an initial graph neural network model can be established first. Afterwards, the initial graph neural network model is trained with the person-image heterogeneous graph samples of the labeled data to obtain the graph neural network model. When determining the type of each social circle, the person-image heterogeneous graph is input into the graph neural network model to obtain the feature vector of each person node and image node.
[0273] Among them, the character-image heterogeneous graph includes information of each character node and information of image nodes. As mentioned above, the information of the character node may include the character's gender, character's age, etc. The feature vector of the character node extracted by the graph neural network model may include the character's gender feature vector, the character's age feature vector, etc. Optionally, the value of the character's gender feature vector is 0 or 1, and optionally, a value of 0 indicates that the gender is male, and a value of 1 indicates that the gender is female. The value range of the character's age feature vector can be [0,100], and the value of the character's age can be an integer.
[0274] As described above, the information of the image node may include the shooting time information of the image, the shooting location of the image, whether it is shot by the front camera, etc. The feature vector of the image node extracted by the graph neural network model may include: shooting time feature vector, shooting location feature vector and whether it is a front camera feature vector, etc. Among them, the shooting time feature vector may include shooting time feature vector, time information feature vector, month difference feature vector and week feature vector, etc. Specifically, the time information feature vector refers to the feature vector of the time information at the shooting time. The value range of the time information feature vector may be [0,23], and the value of the time information feature vector is an integer. The month difference feature vector refers to the feature vector of the difference between the month information at the shooting time and the month information at the current time. The value range of the month difference feature vector may be [0,100], and the value of the month difference feature vector is an integer. The week feature vector refers to the feature vector of the week information at the shooting time. The value range of the week feature vector may be [1,7], and the value of the week feature vector is an integer. The shooting location feature vector may include a one-hot encoding vector defined by home, company, scenic spot, etc. The value of the feature vector of whether the front camera is used can be 0 or 1, which is optional. A value of 0 indicates that the image is not taken with the front camera, and a value of 1 indicates that the image is taken with the front camera.
[0275] Optionally, the graph neural network model can be a modeling relational data with graph convolutional networks (R-GCN) model. The main idea of extracting feature vectors in the R-GCN model is that the feature vector of the l+1th layer of node i can be obtained by aggregating the feature vectors of the lth layer of neighboring nodes of different relationship types. In order to distinguish and learn the data of each relationship, the neighboring nodes of different relationships are transformed with different transformation matrices (also called weight matrices).
[0276] In a specific embodiment, the calculation formula of the R-GCN model can be as follows:
[0277]
[0278] in, Represents the feature vector of the l+1th layer of node i. represents the set of neighbor nodes whose relationship with node i is r. For the person-image heterogeneous graph, there is only one relationship r, which is the inclusion relationship between person and image. i,r represents the regularization constant. Optional, c i,r The value of can be set according to the requirements, for example, c i,r The value of can be Express Perform determinant calculations. Represents the transformation matrix of the lth layer of relation r. For neighbor nodes of the same relation (i.e., the same type of edge), the same transformation matrix is used Optional, It can be a linear conversion function.
[0279] 2) Input the feature vectors of each character node and image node into the pre-trained multi-layer perceptron model to obtain the relationship type between each character node and the central character node (referred to as the relationship type of the character node for short).
[0280] The multi-layer perceptron model is used to predict the type of relationship between each character node and the central character node. The type of relationship between the character node and the central character node may include but is not limited to family members, colleagues, and classmates.
[0281] Optionally, an initial multi-layer perceptron model may be established first. Then, the initial multi-layer perceptron model is trained by node feature vector samples of labeled data to obtain a multi-layer perceptron model. When determining the type of a social circle, the feature vectors of the person node and the image node obtained in step 1) are input into the multi-layer perceptron model to obtain the relationship type of each person node and the central person node, that is, to obtain the relationship type of each person node.
[0282] 3) Count the number of people corresponding to each relationship type in each social circle, and determine the relationship type with the largest number of people as the type of the social circle.
[0283] For example, for any social circle C, the relationship types of the characters in the social circle C include "colleague" and "classmate", then the number of characters with the relationship type of "colleague" in the social circle C is counted (for example, x1), and the number of characters with the relationship type of "classmate" in the social circle C is counted (for example, x2). If x1 is greater than x2, "colleague" is determined as the type of the social circle C.
[0284] 4) In each social circle, the relationship type of the characters that are different from the type of the social circle (ie, the nodes corresponding to the characters to be modified) is modified to the type of the social circle.
[0285] For example, in the above example, if the type of social circle C is determined to be "colleague", the relationship type of each character in social circle C whose relationship type is not "colleague" is modified to "colleague". In this way, the relationship type of the characters in the social circle C that are judged to be biased can be corrected, so that the relationship between the characters in the determined social circle C and the central character is consistent and accurate, which is convenient for subsequent generation of albums according to social circles and generation of album themes according to social circle types.
[0286] For example, Fig.13 Based on Fig.10 The decentralized character connection diagram is a schematic diagram of the recognition results obtained by identifying social circles and their types. Fig.13 As shown, in this embodiment, the identified social circles include 2, and the types of these 2 social circles are family and colleagues. Based on this, the relationship type between each character in the "family" social circle and the central character (shown as black solid color filling in the figure) is corrected to "family" (shown as diagonal pattern filling in the figure), and the relationship type between each character in the "colleague" social circle and the central character is corrected to "colleague" (shown as black dot pattern filling in the figure).
[0287] Based on the example described in this embodiment, the step "S108, the album generation module generates an album according to the social circle, and generates the theme of the album according to the type of the social circle" in the above embodiment may specifically include:
[0288] The album generation module determines each person in the social circle C as a target person, selects the image of the target person in the gallery data storage module, and generates a target album. According to the type of the social circle, the theme of the target album is generated. For example, the theme of the generated target album can be "Lovely Colleagues".
[0289] In this embodiment, according to the face clustering result, a person-image heterogeneous graph is generated, and according to the person-image heterogeneous graph, the intimacy between two persons is determined. After that, based on the intimacy and the person-image heterogeneous graph, a person connection graph is generated, and the central person is identified based on the person-image heterogeneous graph. Then, after removing the central person in the person connection graph, community discovery is performed to obtain a social circle. After obtaining the social circle, the type of each social circle is determined based on the identified central person and the person-image heterogeneous graph. In the first aspect, the above process can identify the social circle and the type of social circle based on the face clustering result, without relying on the user's annotation, with high intelligence and improved user experience. In the second aspect, the process can accurately identify the central person without obtaining the user's face unlocking information, and then identify the social circle and the type of social circle, which will not affect the security of the user's private information, and further improve the user experience. In the third aspect, the above process is based on the images in the gallery data storage module, can be completed offline, does not rely on the network, compared with identifying the social circle and the type of social circle through social behavior on the social network, this method is more applicable, and the recognition result is more targeted for the user of the electronic device, thereby further improving the user experience. Fourthly, the above process first identifies the social circle, then determines the social circle type based on the identification result of the social circle, and corrects the relationship type of the characters in the social circle. Compared with simply classifying the relationships between characters to obtain different social circles and social circle types, the method provided by this application has higher accuracy in identifying social circles and social circle types, and provides a better user experience.
[0290] It should be noted that the order of the method provided in the embodiment of the present application is only an example and does not constitute any limitation on the album generation method. In practical applications, there can be multiple orders between the steps in accordance with the implementation logic. For example, two steps can be executed one after the other or simultaneously. And when they are executed one after the other, in accordance with the implementation logic, the order between the two steps is not limited. For example, in the above embodiment, steps S202 to S204 and step S205 can be performed first, and then step S205, or step S205 can be performed first, and then steps S202 to S204, or step S205 can be performed while steps S202 to S204 are being executed.
[0291] The process of determining the intimacy between two characters is described below.
[0292] In a specific embodiment, taking any two persons, person a and person b, included in the images of the co-occurrence image set as an example, the intimacy calculation process of person a and person b is described. In the implementation process of the above step "S203, determining the intimacy between the two persons in the face clustering result according to the image information of each image in the co-occurrence image set", the intimacy between the two persons included in the images of the co-occurrence image set is determined according to the image information of each image in the co-occurrence image set, including:
[0293] 1) Calculate the mutual information between person a and person b.
[0294] Optionally, the mutual information PMI(a,b) between person a and person b can be calculated by formula (4):
[0295]
[0296] Among them, pic a Indicates the number of co-occurrence images containing person a in the co-occurrence image set. b Indicates the number of co-occurrence images containing person b in the co-occurrence image set. a,b Indicates the number of co-occurrence images of characters a and b in the co-occurrence image set. total_pics indicates the total number of all co-occurrence images in the co-occurrence image set.
[0297] In one embodiment, the intimacy between person a and person b may be directly represented by PMI(a, b), that is, the value of PMI(a, b) is the intimacy between person a and person b.
[0298] In other embodiments, the intimacy between person a and person b may be determined based on PMI (a, b) in combination with other parameters. As a possible implementation, the intimacy between person a and person b may be determined based on PMI (a, b) in combination with the time distribution divergence and location distribution divergence of the image according to the following process. Specifically, on the basis of the above step 1), the following steps 2) to 6) may also be included:
[0299] 2) According to the shooting time of the co-occurrence images of the characters ab, the time distribution of the co-occurrence images of the characters ab (hereinafter referred to as the co-occurrence time distribution of the characters ab) is counted.
[0300] The co-occurrence time distribution of characters ab is used to characterize the distribution of the shooting moments of the co-occurrence images of characters ab in multiple preset time intervals. The interval length and number of the preset time intervals can be determined according to actual conditions.
[0301] As a possible implementation, a day (00:00 to 24:00) can be divided into multiple preset time intervals based on the timing information, wherein the duration of each preset time interval is less than 24 hours. In a specific embodiment, a day can be divided into 24 preset time intervals, and the duration of each preset time interval is 1 hour. For example, the 24 preset time intervals may include: [00:00, 01:00), [01:00, 02:00), [02:00, 03:00) ... [23:00, 24:00).
[0302] It should be noted that the preset time interval can also be divided in other ways. For example, a month (00:00 on the 1st to 24:00 on the 31st) can be divided into multiple preset time intervals based on day information and timing information. This application does not impose any restrictions on the division method of the preset time interval. The following is an example of dividing a day into multiple preset times based on timing information. The execution process of the method under other division methods is similar to this and will not be repeated.
[0303] In a specific embodiment, the co-occurrence time distribution of characters ab may be counted according to the following process:
[0304] a. Count the number of matching co-occurrence images in each preset time interval among the co-occurrence images of person a and b.
[0305] Among them, the matching co-occurrence image within a preset time interval refers to: in the co-occurrence image of the characters ab, the timing information belongs to the preset time interval (that is, the shooting time matches the preset time interval). For example, the matching image within the preset time interval [08:00, 09:00) refers to the co-occurrence image of the characters ab, the timing information belongs to [08:00, 09:00). For example, the shooting time of a certain character ab co-occurrence image x (referred to as image x) is 08:18:12 on April 12, 2023, and the timing information of image x is 08:18:12, which belongs to [08:00, 09:00), so image x is the matching co-occurrence image of the preset time interval [08:00, 09:00). According to this method, the matching co-occurrence images within each preset time interval are determined, and the number of matching co-occurrence images within each preset time interval is obtained by counting.
[0306] Optionally, during the statistical process, for multiple matching co-occurrence images with the same date information within the same preset time interval, the number can be counted only once. That is: the co-occurrence images of characters ab with the same date information and timing information belonging to the same preset time interval are counted as one image. For example, the shooting time of image x is 08:18:12 on April 12, 2023, and the shooting time of character ab co-occurrence image y (referred to as image y) is 08:25:10 on April 12, 2023. The date information of image x and image y is the same, both April 12, 2023. The timing information of image x and image y both belong to the preset time interval [08:00, 09:00), which are matching co-occurrence images within the preset time interval. Therefore, when counting, image x and image y are regarded as one image, and the number is only counted once. This is to take into account that the co-occurrence time distribution of characters ab will be used to determine the divergence (i.e., uniformity) of the time distribution of characters ab in the future, and the intimacy between characters a and b is reflected by the time distribution divergence. The more evenly the time distribution is, the closer the characters A and B are. In actual use, even if two very close characters are taking photos with a camera, they may take photos together in a preset time interval on a certain day, resulting in multiple images. However, adding concentrated photos taken in a preset time interval to the time distribution statistics will result in an uneven distribution of the co-occurrence time of the two characters, which may cause the intimacy of two people with a closer relationship to be lower, while the intimacy of two people with a less intimate relationship who did not take concentrated photos will increase. Therefore, the method provided by this implementation method counts the number of images taken in concentrated photos only once, and eliminates images produced by concentrated photos, so as to prevent affecting the accuracy of the co-occurrence time distribution statistics of characters A and B, thereby improving the accuracy of intimacy calculation.
[0307] In a specific embodiment, taking multiple preset time intervals including: [00:00, 01:00), [01:00, 02:00), [02:00, 03:00) ... [23:00, 24:00) as an example, the process of counting the number of matching co-occurring images in each preset time interval can be expressed by the following formula:
[0308] By vector Indicates the number of matching co-occurrence images in each preset time interval. ab [i] represents vector t ab The element i in represents the serial number of the preset time interval, i = 1, 2, 3 ... 24. When i = 1, the preset time interval 1 can be [00:00, 01:00), when i = 2, the preset time interval 2 can be [01:00, 02:00), when i = 3, the preset time interval 3 can be [02:00, 03:00) ... when i = 24, the preset time interval 24 can be [23:00, 24:00). t ab [i] represents the number of matching images within the preset time interval i.
[0309] First, the vector t ab Each element in is initialized to 0. Then, the number of matching co-occurrence images in each preset time interval is counted based on formula (5):
[0310]
[0311] Wherein, m represents the co-occurrence image m of person ab (abbreviated as image m). Function f(m) represents the preset time interval to which the timing information of image m belongs. ab [f(m)] represents the number of matching images within the preset time interval f(m). g(m) = 1 means that the co-occurrence images of person ab with the same date information as image m and the same preset time interval have been counted once. In this case, they are no longer counted. Therefore, t ab The value of [f(m)] remains unchanged, that is, t ab [f(m)]=t ab [f(m)] Correspondingly, g(m) = 0 means that the co-occurrence image with the same date information as image m and the same preset time interval has not been counted. In this case, the image needs to be included in the statistics. Therefore, t ab The value of [f(m)] is increased by 1, that is, t ab [f(m)]=t ab [f(m)]+1.
[0312] M represents a set of co-occurring images of characters ab, M = {M1, M2, ..., Mx}. M1, M2 ... Mx all represent co-occurring images of characters ab, and x represents the total number of co-occurring images of characters ab.
[0313] It can be understood that, when multiple preset time intervals include: [00:00, 01:00), [01:00, 02:00), [02:00, 03:00) ... [23:00, 24:00), when judging which preset time interval the timing information of a certain image belongs to, it can be judged only by the hour information of the image, without obtaining minute information and second information, etc. For example, if the hour information of image m is 10 hours, then image m belongs to the preset time interval [10:00, 11:00). Therefore, in this embodiment, the multiple preset time intervals are divided into 24 preset time intervals according to the above method, and the lower limit of each preset time interval is an integer number of hours, which can reduce the amount of calculation when calculating time distribution statistics and improve the efficiency of algorithm operation.
[0314] b. Determine the co-occurrence time distribution of characters ab according to the number of matching co-occurrence images in each preset time interval.
[0315] Based on the number of matching co-occurrence images in each preset time interval determined above, the proportion of matching co-occurrence images in each preset time interval in all co-occurrence images of characters ab is determined to obtain the co-occurrence time distribution of characters ab.
[0316] For the sake of ease of description, the proportion of the number of matching co-occurrence images within a preset time interval in all co-occurrence images of characters ab is referred to as the co-occurrence image proportion corresponding to the preset time interval.
[0317] Optionally, the process of determining the co-occurrence time distribution of characters ab can be expressed by formula (6):
[0318]
[0319]
[0320] Among them, the vector represents the co-occurrence time distribution of characters a and b, d ab,t [i] is any element in the vector, d ab,t [i] represents the proportion of co-occurring images corresponding to the preset time interval i.
[0321] As an example, the co-occurrence time distribution graph of characters a and b can be as follows: Fig.14 The ordinate represents the proportion of co-occurring images corresponding to the preset time interval (abbreviated as proportion in the figure), and the abscissa represents the sequence number of the preset time interval, that is, the value of i. Fig.14The co-occurrence time distribution of characters ab shown in can be expressed as:
[0322] 3) Determine the KL divergence between the co-occurrence time distribution and the uniform time distribution of character ab, and obtain the temporal KL divergence of character ab.
[0323] Uniform time distribution means that all co-occurrence images of characters ab are evenly distributed in multiple preset time intervals, that is, the proportion of co-occurrence images corresponding to each preset time interval is the same, which is 1 / the number of preset time intervals. For example, when the number of preset time intervals is 24, uniform time distribution means that the proportion of co-occurrence images corresponding to each preset time interval is 1 / 24.
[0324] The temporal KL divergence of character ab is used to characterize the difference between the co-occurrence time distribution of character ab and the uniform time distribution. In other words, the temporal KL divergence of character ab is used to characterize the uniformity of the distribution of the co-occurrence images of character ab in multiple preset time intervals, thereby reflecting the uniformity of the co-occurrence of character a and character b in time.
[0325] The time distribution can be expressed as a vector The elements in this set are all 1 / 24. Then the temporal KL divergence of character ab can be calculated by formula (7):
[0326]
[0327] Among them, D ab,t represents the temporal KL divergence of character ab, d u,t [i] represents vector d u,t Any element in d u,t [i]=1 / 24.
[0328] As an example, a comparison diagram of the co-occurrence time distribution and uniform time distribution of characters a and b can be shown as follows: Fig.15 The meanings of the horizontal and vertical axes in the figure are the same as Fig.14 The same, no further elaboration.
[0329] 4) Count the location distribution of the co-occurrence images of characters ab (hereinafter referred to as the co-occurrence location distribution of characters ab).
[0330] The co-occurrence location distribution of characters ab is used to characterize the distribution of the shooting locations of the co-occurrence images of characters ab in multiple known locations. The known locations can be obtained through the shooting locations of the images in the gallery data storage module.
[0331] In a specific embodiment, the co-occurrence location distribution of characters ab can be counted according to the following process:
[0332] a. Determine a set of shooting locations, which includes the shooting locations of all images in the gallery data storage module.
[0333] Optionally, the set of shooting locations can be represented as N1, N2, ..., Ny are elements of the set N, representing the shooting locations. For example, the shooting location set y represents the total number of shooting locations.
[0334] Optionally, at any time before executing step a, by searching for the shooting locations of all images stored in the gallery data storage module, a shooting location set can be generated and stored in the gallery data storage module. When the images in the gallery data storage module are updated, the shooting location set can be updated so that the shooting location set can be obtained as needed when executing the method provided in the embodiment of the present application.
[0335] For the convenience of description, the elements in the shooting location set are referred to as location elements in the following. That is, the shooting location set includes multiple location elements, and each location element is a shooting location of the image in the gallery data storage module.
[0336] b. Count the number of co-occurring images of characters ab corresponding to each location element.
[0337] For any location element Ni, the number of co-occurrence images of people ab corresponding to the location element refers to the number of co-occurrence images of people ab taken at location Ni. For example, the number of co-occurrence images of people ab corresponding to the location element "Beijing" refers to the number of co-occurrence images of people ab taken at location "Beijing".
[0338] c. Determine the co-occurrence location distribution of characters ab based on the number of co-occurrence images of characters ab corresponding to each location element.
[0339] Based on the number of co-occurrence images of the characters ab corresponding to each location element determined above, the proportion of the number of co-occurrence images of the characters ab corresponding to each location element in all co-occurrence images of the characters ab is determined to obtain the co-occurrence location distribution of the characters ab.
[0340] For the convenience of description, the proportion of the number of co-occurrence images of characters ab corresponding to a certain place element in all co-occurrence images of characters ab is referred to as the proportion of co-occurrence images corresponding to the place element.
[0341] Optionally, when determining the proportion of co-occurrence images corresponding to each location element, if the number of co-occurrence images of characters ab corresponding to the location element is 0, the proportion is determined to be 0; if the number of co-occurrence images of characters ab corresponding to the location element is not 0 (i.e., greater than 0), the proportion of co-occurrence images corresponding to the location element is determined to be 1 / the number of non-zero location elements. A non-zero location element refers to a location element whose number of corresponding co-occurrence images of characters ab is not 0. For example, the shooting location set includes 10 location elements. According to statistics, the number of co-occurrence images of characters ab corresponding to the five location elements N1, N3, N5, N7 and N10 in the set is not 0, so the proportion of co-occurrence images corresponding to these five location elements is 1 / 5=0.2, and the proportion of co-occurrence images corresponding to other location elements is 0. Then, the co-occurrence location distribution of characters ab can be expressed as:
[0342] As another optional implementation, in the process of counting the location distribution of co-occurrence images of characters ab, the above steps b and c can also be replaced by: counting whether co-occurrence images of characters ab corresponding to each location element exist; and determining the co-occurrence location distribution of characters ab based on the statistical results.
[0343] Optionally, the tag set includes a tag value corresponding to each place element, and the tag value is used to mark whether there is an image with the shooting location of the place element in the set of co-occurring images of the characters ab. Specifically, for any place element Ni, determine whether there is an image with the shooting location Ni in the set of co-occurring images of the characters ab; if so, the tag value corresponding to the place element Ni is determined to be 1 (that is, the first value); if not, the tag value corresponding to the place element Ni is determined to be 0 (that is, the second value). Afterwards, the total number of place elements with a tag value of 1 is counted to obtain the number of non-zero place elements (that is, the total number of the first value). Then, the proportion of co-occurring images corresponding to the place elements with a tag value of 1 is set to 1 / the number of non-zero place elements.
[0344] Specifically, the above process can be expressed as follows through the formula:
[0345] By vector Represents the set of tag values corresponding to each location element. ab [j] represents vector l ab The element in the , j is the number of the element in position, j = 1, 2, 3...y. ab [j] represents the tag value corresponding to location element j.
[0346] First, the vector l ab Each element in is initialized to 0. Then, based on formula (8), vector l abAssign values to each element:
[0347]
[0348] Where m represents the co-occurrence image m of person ab (abbreviated as image m). Function h(m) represents the shooting location of image m. ab [h(m)] represents the tag value corresponding to the shooting location h(m). q(m) = 0 means that the shooting location h(m) has not been counted, and the tag value of the location element with the same shooting location h(m) is currently 0. In this case, since the shooting location of image m is h(m), the tag value corresponding to the location element with the same shooting location h(m) is assigned to 1, that is, l ab [h(m)] = 1. Correspondingly, q(m) = 1 means that the shooting location h(m) has been counted once, that is, it has been determined that there is a co-occurrence image of person ab with the shooting location h(m), and the tag value of the location element with the same shooting location h(m) has been assigned 1. In this case, l ab The value of [h(m)] remains unchanged, that is, l ab [h(m)]=l ab [h(m)] M represents the set of co-occurring images of person ab, which will not be described in detail.
[0349] Then, the co-occurrence location distribution of characters ab is determined by formula (9):
[0350]
[0351]
[0352] Among them, the vector represents the co-occurrence location distribution of characters a and b, d ab,l [j] is any element in the vector, d ab,l [j] represents the proportion of co-occurrence images corresponding to location element j.
[0353] In this implementation, the co-occurrence location distribution of characters ab will be used to determine the divergence (i.e., uniformity) of the location distribution of characters ab in the future, and the intimacy between characters a and b is reflected by the location distribution divergence. The more uniform the location distribution, the closer the characters a and b are. In actual use, even two very close characters may take photos together at a certain location when taking photos with a camera, resulting in multiple images. Adding concentrated photos at a certain location to the location distribution statistics will result in an uneven distribution of the co-occurrence locations of the two characters, which may make the intimacy of two people with a closer relationship lower, while the intimacy of two people with a less intimate relationship who did not take concentrated photos will increase. Therefore, in this implementation, by counting whether there are co-occurrence images of characters ab corresponding to each location element, the location distribution of characters ab is analyzed, without considering the number of images taken at each location element, to prevent the images produced by concentrated photos from affecting the accuracy of the co-occurrence location distribution statistics of characters ab, thereby improving the accuracy of intimacy calculation.
[0354] As an example, the co-occurrence locations of characters ab are distributed as follows: When , the corresponding co-occurrence location distribution map can be shown as Fig.16 As shown in the figure, the ordinate represents the proportion of co-occurring images corresponding to the location element, and the abscissa represents the sequence number of the location element, that is, the value of j.
[0355] 5) Determine the KL divergence of the co-occurrence location distribution and the uniform location distribution of character ab, and obtain the location KL divergence of character ab.
[0356] Uniform location distribution means that all co-occurrence images of all characters ab are evenly distributed among all location elements in the shooting location set, that is, the proportion of co-occurrence images corresponding to each location element is the same, which is 1 / the total number of location elements. For example, when the total number of location elements is 10, uniform location distribution means that the proportion of co-occurrence images corresponding to each location element is 1 / 10=0.1.
[0357] The location KL divergence of person ab is used to characterize the difference between the co-occurrence location distribution of person ab and the uniform location distribution. In other words, the location KL divergence of person ab is used to characterize the uniformity of the distribution of the co-occurrence images of person ab in multiple preset time intervals, thereby reflecting the uniformity of the co-occurrence of person a and person b in location.
[0358] The locations are distributed and can be represented as vectors All elements in this set are 1 / y. Then the KL divergence of the location of character ab can be calculated by formula (10):
[0359]
[0360] Among them, D ab,lrepresents the KL divergence of the location of character ab, d u,l [j] represents vector d u,l Any element in d u,l [j] = 1 / y.
[0361] As an example, a comparison diagram of the co-occurrence location distribution and uniform location distribution of characters a and b can be shown as follows: Fig.17 The meanings of the horizontal and vertical axes in the figure are the same as Fig.16 The same, no further elaboration.
[0362] 6) Determine the intimacy between person a and person b based on the time KL divergence of person ab, the location KL divergence of person ab, and PMI(a,b).
[0363] In a specific embodiment, the intimacy between character a and character b can be determined according to formula (11):
[0364] closure(a,b)=PMI(a,b)×(1-D ab,t )×(1-D ab,t ) (11)
[0365] Among them, closure(a,b) represents the intimacy between character a and character b, D ab,t represents the time KL divergence of character ab, D ab,l Indicates the KL divergence of the location of person ab.
[0366] It can be seen from formula (11) that D ab,t The smaller it is, the larger the closure(a,b); D ab,l The smaller it is, the larger the closure(a,b). ab,t The smaller it is, the more uniform the co-occurrence time distribution of characters a and b is; D ab,l The smaller it is, the more uniform the distribution of the co-occurrence locations of characters ab is. That is, according to formula (11), the more uniform the co-occurrence time distribution of characters ab is, the greater the intimacy between characters a and b is, and the closer characters a and b are; moreover, the more uniform the co-occurrence time and location of characters ab is, the greater the intimacy between characters a and b is, and the closer characters a and b are.
[0367] In this embodiment, when determining the intimacy between person a and person b, on the basis of PMI(a,b), the co-occurrence time KL divergence and co-occurrence location KL divergence of person ab are further added. The co-occurrence time KL divergence of person ab can characterize the uniformity of the temporal distribution of the co-occurrence images of person a and person b. The co-occurrence location KL divergence of person ab can characterize the uniformity of the location distribution of the co-occurrence images of person a and person b. The closer the social relationship between person a and person b, the more uniform the distribution of their co-occurrence images in time and location. Therefore, in this embodiment, the co-occurrence time KL divergence and co-occurrence location KL divergence of person ab are integrated into the calculation of intimacy, which improves the accuracy of intimacy calculation, thereby improving the accuracy of social circle recognition.
[0368] It can be understood that the above steps 2) to 5) are also called divergence calculation operations, and the above steps 1) to 6) are also called intimacy calculation operations.
[0369] The following is an explanation of the method for identifying the central person.
[0370] In a specific embodiment, in the above step S205, identifying the central person according to the person identification-heterogeneous graph may specifically include:
[0371] 1) According to the person-image heterogeneous graph, determine the image set of each person.
[0372] As described in the above embodiment, an image containing a certain person may be referred to as an image of the person.
[0373] Specifically, according to the person-image heterogeneous graph, all image nodes connected to a person node can be determined, and a set of images corresponding to these image nodes can be generated, that is, the image combination of the person of the person node can be obtained. Taking any person a as an example, the connection matrix can be determined according to the connection relationship matrix shown in Table 4, and the image nodes including the person node a in the connection matrix can be determined: image node A, image node C, and image node N, etc. The images corresponding to these image nodes are generated into a set to obtain the image set of person a.
[0374] 2) According to the image information of each image in the image set of person a, determine the time KL divergence of the image of person a (hereinafter referred to as the time KL divergence of person a) and the location KL divergence of the image of person a (hereinafter referred to as the location KL divergence of person a).
[0375] Wherein, person a is any person in the person-image heterogeneous graph. That is to say, for each person in the person-image heterogeneous graph, its time KL divergence and location KL divergence are determined according to the method in this step, thereby obtaining the time KL divergence and location KL divergence of each person.
[0376] In a specific embodiment, the time KL divergence and location KL divergence of person a may be determined according to the following process:
[0377] a. According to the shooting time of each image in the image set of person a, the time distribution of the images of person a (hereinafter referred to as the time distribution of person a) is counted.
[0378] The time distribution of person a is used to characterize the distribution of the shooting moments of the image of person a in a plurality of preset time intervals. The preset time intervals are similar to those in the above embodiment and will not be described in detail.
[0379] Counting the time distribution of person a is similar to counting the co-occurrence time distribution of person ab in the above embodiment, except that: in this embodiment, the scope of statistics is the image set of person a, while in the above embodiment, the scope of statistics is all co-occurrence images of person a, and the object of statistics in this embodiment is the image of person a, while in the above embodiment, the object of statistics is the co-occurrence image of person ab. In this embodiment, in the image set of person a, the image whose shooting time matches a preset time interval can be called the matching image of the preset time interval. In the above embodiment, the image whose shooting time matches a preset time interval can be called the matching co-occurrence image of the preset time interval. The specific implementation process and beneficial effects of this embodiment refer to the above embodiment and will not be repeated.
[0380] b. Determine the KL divergence between the time distribution of character a and the uniform time distribution, and obtain the time KL divergence of character a.
[0381] The uniform time distribution is the same as in the above embodiment. The temporal KL divergence of character a is used to characterize the difference between the temporal distribution of character a and the uniform time distribution. In other words, the temporal KL divergence of character a is used to characterize the uniformity of the distribution of the image of character a in multiple preset time intervals, that is, to reflect the uniformity of the appearance of character a in time.
[0382] The calculation method of the temporal KL divergence of character a is similar to the calculation method of the temporal KL divergence of character ab, and will not be repeated here.
[0383] c. According to the shooting location of each image in the image set of person a, the location distribution of the images of person a (hereinafter referred to as the location distribution of person a) is counted.
[0384] The location distribution of person a is used to represent the distribution of shooting locations of images of person a.
[0385] In a specific embodiment, the co-occurrence location distribution of characters ab can be counted according to the following process:
[0386] The location distribution of person a is similar to the location distribution of person ab in the above embodiment, except that in this embodiment, the scope of statistics is the image set of person a, not the image set of person ab co-occurrence, and the object of statistics is the image of person a, not the image of person ab co-occurrence. The specific implementation process and beneficial effects of this process are shown in the above embodiment, and will not be repeated here.
[0387] d. Determine the KL divergence between the location distribution of person a and the uniform location distribution, and obtain the location KL divergence of person a.
[0388] The uniform location distribution is the same as in the above embodiment. The location KL divergence of person a is used to characterize the difference between the location distribution of person a and the uniform location distribution. In other words, the location KL divergence of person a is used to characterize the uniformity of the distribution of the image of person a in multiple preset time intervals, thereby reflecting the uniformity of the appearance of person a in the location.
[0389] The calculation method of the location KL divergence of character a is similar to the calculation method of the location KL divergence of character ab, and will not be repeated here.
[0390] 3) Determine the central character based on the time KL divergence and location KL divergence of all characters.
[0391] In one embodiment, the time KL divergence and the location KL divergence of each character can be summed to obtain the divergence sum of each character. Afterwards, the divergence and ranking of all characters (i.e., the first ranking) are taken, and the Q1 characters with the smallest divergence and are taken as the central characters, where Q1 is an integer greater than or equal to 1. If the ranking is in the order of the divergence and ranking from small to large, the top Q1 characters (TOP Q1) in the divergence and ranking are taken as the central characters. If the ranking is in the order of the divergence and ranking from large to small, the last Q1 characters (END Q1) in the divergence and ranking are taken as the central characters. It can be understood that when summing the KL divergence, it can be a direct summation or a weighted summation, and the embodiments of the present application do not impose any restrictions on this.
[0392] In another embodiment, the time KL divergence of all characters can be ranked in order from small to large, and the location KL divergence can be ranked in order from small to large. Then, the time KL divergence ranking and location KL divergence ranking of each character are summed to obtain the divergence ranking sum corresponding to each character. The Q2 characters with the smallest sum of all divergence rankings are taken as the central characters, and Q2 is an integer greater than or equal to 1. For example, the time KL divergence ranking of characters A, B, and C is {1,2,3}, and the location KL divergence ranking is {2,3,1}. Then, the two rankings are summed to obtain the divergence ranking sum of characters A, B, and C: {3,5,4}. If Q2 is 1, the character A with the smallest divergence ranking sum is taken as the central character. It should be noted that when determining the central character, if the Q2 central characters cannot be directly obtained because the divergence ranking sums of multiple characters are the same, the character with the smaller time KL divergence and / or location KL divergence is selected from the multiple characters as the central character. In addition, when the rankings are summed, it can be a direct summation or a weighted summation, and the embodiments of the present application do not impose any limitation on this.
[0393] That is to say, the person whose time KL divergence and location KL divergence meet the first preset condition is determined as the central person, wherein the first preset condition is any one of the following conditions: ① The divergence and ranking are in the top Q1 of the first ranking, and the first ranking is obtained by ranking the characters in order of divergence and ranking from small to large; ② The divergence and ranking are in the top Q2 of the second ranking, and the second ranking is obtained by ranking the characters in order of divergence and ranking from small to large; ③ The divergence and ranking are in the last Q1 of the fourth ranking, and the fourth ranking is obtained by ranking the characters in order of divergence and ranking from large to small; ④ The divergence and ranking are in the last Q2 of the fifth ranking, and the fifth ranking is obtained by ranking the characters in order of divergence and ranking from large to small.
[0394] In practical applications, in addition to generating albums based on images corresponding to people in a social circle, there are also multiple ways to generate albums, for example, albums can be generated based on one or more of the central person, the intimacy between people, time, location, etc. Several specific implementations are provided below.
[0395] A. Generate an album based on the intimacy between the central person and the social circle (also known as the circle intimacy of the social circle)
[0396] In one embodiment, after the above step S207, the album generation method may further include: determining the average of the intimacy between the central person and the people in each social circle, and obtaining the intimacy between the central person and each social circle.
[0397] Specifically, after identifying the social circle, based on the character connection graph, the characters related to the central character in each social circle can be determined, and the intimacy between these characters and the central character can be determined. Afterwards, for each social circle, the sum of the intimacy between each character in the social circle and the central character is calculated, and the sum of the intimacy is averaged, and the average is used as the intimacy between the central character and the social circle. In this way, the intimacy between the central character and each social circle can be determined. It can be understood that the intimacy between the characters in the social circle who are not related to the central character and the central character is 0.
[0398] In a specific embodiment, the intimacy between the central person and any social circle C can be determined by formula (3):
[0399]
[0400] Among them, closure C Indicates the intimacy between the central person and the social circle C. C_num indicates the number of people in the social circle C. The central person is represented as center. closure(i,center) indicates the intimacy between any person i in the social circle C and the central person center. The specific process of calculating the intimacy between person i and the central person center may be the same as step S203.
[0401] Generate an album based on the intimacy between the central person and each social circle. In a specific embodiment, in step S108 of the above embodiment, "the album generation module generates an album based on the social circle" may also include: the album generation module generates an album based on the images of the characters in the intimate social circle. Among them, the intimate social circle refers to the social circle whose intimacy with the central person is greater than the preset threshold, or the intimate social circle may be the X social circles with the greatest intimacy with the central person, where X is a positive integer. In other words, the social circle whose intimacy meets the second preset condition is determined as the intimate social circle, wherein the second preset condition may be: the intimacy of the social circle is greater than the preset threshold, or the intimacy ranking of the social circle is in the top X positions of the third ranking, and the third ranking is obtained by ranking the at least one social circle in order from large to small according to the intimacy of the circle, where X is a positive integer. Optionally, the album generated based on the images of the characters in the intimate social circle may be based on the content such as "the closest person" as the theme of the album. In this implementation, the album is generated based on the intimate social circle, which further fits the actual social situation of the user, thereby further improving the user experience.
[0402] B. Generate photo albums based on central characters
[0403] As described above, the number of central characters can be one or more. When the number of central characters is one, the images of the central character stored in the gallery data storage module can be filtered according to preset filtering conditions to obtain the photo album of the central character. The preset filtering conditions may include, for example: the aesthetic score of the image, the shooting time of the image, and the number of characters contained in the image, etc. Among them, when generating a photo album, deduplication processing can be adopted, that is, only a preset number (for example, 1) of multiple images with the same or similar image content are selected to improve the effect of generating the photo album and improve the user experience. Deduplication processing can also be adopted in other embodiments of generating photo albums in the present application, which will not be repeated.
[0404] When there are multiple central persons, the central person with the smallest time distribution divergence and location distribution divergence can be selected as the "most central person" based on the divergence sum or divergence ranking sum of the multiple central persons, and an album of the most central person can be generated.
[0405] In addition, an album of a person corresponding to preset annotation information (eg, "I") may be generated based on the user's annotation.
[0406] C. Generate photo albums based on close people
[0407] "Close characters" refer to characters whose intimacy with the central character meets preset conditions. For example, the characters ranked in the top S in terms of intimacy with the central character. Specifically, the intimacy between each character and the central character can be determined, and the intimacy can be ranked, and the top S characters ranked can be regarded as close characters. The images containing close characters in the gallery data storage module are filtered according to the preset filtering conditions to obtain the album of close characters.
[0408] Here are a few more examples of generating album themes:
[0409] In one embodiment, the theme of the album can be selected from a preset theme library according to the basis for generating the album (i.e., generating the album based on the social circle, the intimacy between the central person and the social circle, the central person or close persons, etc.), preset filtering conditions, etc.
[0410] For example, if the album is generated based on the central character and the preset filtering condition includes "the image was taken one year ago", the theme of the album can be selected from the first theme library. The first theme library can include themes such as "Time Flies", "You Once Were There", and "Flowing Years".
[0411] For another example, if the photo album is generated based on a close person, the preset screening condition includes "the image was taken n (n is greater than 1) years ago", and the annotation information of the close person includes "Zhang San", then the theme of the photo album can be selected from the second theme library. The second theme library can include themes such as "Zhang San's Years" and "Zhang San's Past Times".
[0412] For another example, the album is generated based on close people, and the preset screening conditions include "the image was taken within the preset time period" and "the number of people is 1". In other words, the album only contains close people, and the close person is identified as a child, then the theme of the album can be selected from the third theme library. The third theme library can include themes such as "innocent time", "good childhood", and "childlike time".
[0413] The above describes in detail an example of the album generation method provided by the embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.
[0414] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0415] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0416] The electronic device provided in this embodiment is used to execute the above-mentioned album generation method, and thus can achieve the same effect as the above-mentioned implementation method.
[0417] In the case of an integrated unit, the electronic device may further include a processing module, a storage module and a communication module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device to execute stored program codes and data, etc. The communication module may be used to support the electronic device to communicate with other devices.
[0418] Among them, the processing module can be a processor or a controller. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.
[0419] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a Figure 2 Device of the structure shown.
[0420] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the album generation method of any of the above embodiments.
[0421] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the album generation method in the above-mentioned embodiment.
[0422] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory so that the chip executes the album generation method in the above-mentioned method embodiments.
[0423] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.
[0424] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0425] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0426] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0427] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0428] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0429] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for generating a photo album, the method being executed by an electronic device, It is characterized in that The method comprises: Determine a person image isomeric map according to a plurality of images in a gallery application, wherein the person image isomeric map represents a correspondence between the plurality of images and a plurality of persons, wherein an image including a first person among the plurality of images corresponds to the first person, and the first person is any one of the plurality of persons; Determining the intimacy between every two of the multiple characters according to the character image heterogeneous graph, wherein the intimacy represents the degree of association between the every two characters; According to the intimacy between each two characters and the character image heterogeneous graph, identifying a first social circle, where the first social circle is a set of characters having a social relationship; A first photo album is generated according to the first social circle, wherein the first photo album includes images within the first circle, and the first images within the first circle are images including at least one person in the first social circle.
2. The method according to claim 1, It is characterized in that The character image heterogeneous graph includes a plurality of image nodes and a plurality of character nodes, the plurality of image nodes correspond one-to-one to the plurality of images, and the plurality of character nodes correspond one-to-one to the plurality of characters; The identifying the first social circle according to the intimacy between each two characters and the character image heterogeneous graph includes: The plurality of image nodes in the character image heterogeneous graph are removed, and according to the intimacy between each two characters, the character nodes corresponding to two characters whose intimacy is not 0 are connected through edges to obtain a character connection graph; The first social circle is identified according to the character connection graph.
3. The method according to claim 2, It is characterized in that The step of identifying the first social circle according to the character connection graph includes: According to the character image heterogeneous graph, a central character is identified, and the time distribution divergence and location distribution divergence of the image corresponding to the central character meet a first preset condition, the time distribution divergence is the distribution divergence of the shooting time of the image, and the location distribution divergence is the distribution divergence of the shooting location of the image; Removing the character node corresponding to the central character in the character connection graph to obtain a decentralized character connection graph; Perform community discovery on the decentralized character connection graph to obtain the first social circle.
4. The method according to claim 3, It is characterized in that The character image isomorphic graph also includes shooting time and shooting location of the multiple images. The identifying the central character according to the character image isomorphic graph includes: According to the character image heterogeneous graph, respectively performing divergence calculation on the image corresponding to each of the multiple characters to obtain the time distribution divergence and location distribution divergence of the image corresponding to each of the multiple characters; Determine a person whose time distribution divergence and location distribution divergence of the corresponding image meet the first preset condition as the central person; The divergence calculation includes: According to the shooting time of the image corresponding to the first person, counting the distribution of the image corresponding to the first person in multiple preset time intervals to obtain a first time distribution; Determine a divergence between the first time distribution and a uniform time distribution, and obtain a time distribution divergence of an image corresponding to the first person; According to the shooting location of the image corresponding to the first person, statistics are collected on the distribution of the shooting location of the image corresponding to the first person in a plurality of known locations to obtain a first location distribution, wherein the plurality of known locations include the shooting locations of all images in the gallery application; The divergence between the first location distribution and the uniform location distribution is determined to obtain the location distribution divergence of the image corresponding to the first person.
5. The method according to claim 4, It is characterized in that The step of counting the distribution of the shooting time of the image corresponding to the first person in a plurality of preset time intervals according to the shooting time of the image corresponding to the first person to obtain a first time distribution includes: Counting the number of matching images within each of the preset time intervals, wherein the matching images within the first preset time interval refer to images corresponding to the first person, the shooting time of which matches the first preset time interval, and the first preset time interval is any one of the multiple preset time intervals; The first time distribution is determined according to the number of matching images in each of the preset time intervals.
6. The method according to claim 5, It is characterized in that The plurality of preset time intervals are obtained by dividing a day into a plurality of time intervals; and the counting of the number of matching images in each of the preset time intervals includes: If the timing information in the shooting moment of the first target image belongs to the target preset time interval and the second target image has not been counted, the number of matching images within the target preset time interval is increased by one; the first target image is any one of the images corresponding to the first person, and the second target image is an image corresponding to the first person, whose timing information belongs to the target preset time interval and whose date information is the same as the date information in the shooting moment of the first target image.
7. The method according to claim 5 or 6, It is characterized in that The determining the first time distribution according to the number of matching images in each of the preset time intervals includes: The ratio of the number of matching images in each of the preset time intervals to the total number of images corresponding to the first person is determined respectively to obtain the first time distribution.
8. The method according to any one of claims 4 to 7, It is characterized in that The step of counting the distribution of the images corresponding to the first person in a plurality of known locations according to the shooting locations of the images corresponding to the first person to obtain the first location distribution includes: Determine the tag value corresponding to each of the known locations respectively, the tag value corresponding to the first known location is used to mark whether there is an image corresponding to the first person with the first known location as the shooting location, the first known location being any one of the multiple known locations; The first location distribution is determined according to the label value corresponding to each of the known locations.
9. The method according to claim 8, It is characterized in that The tag value includes a first value and a second value, wherein the first value indicates that there are images corresponding to the first person whose shooting location is the first known location, and the second value indicates that there are no images corresponding to the first person whose shooting location is the first known location; the first location distribution indicates the proportion of each of the known locations; The determining the first location distribution according to the tag value corresponding to each of the known locations includes: The proportion of known places whose marking value is the first value is set to 1 / the total number of the first value, and the proportion of known places whose marking value is the second value is set to 0, to obtain the first place distribution.
10. The method according to any one of claims 4 to 9, It is characterized in that The first preset condition includes: The divergence sum is located in the first Q1 position of the first ranking; the divergence sum is the sum of the time distribution divergence and the location distribution divergence, the first ranking is obtained by ranking the multiple characters in ascending order of the divergence sum, and Q1 is a positive integer; or, The sum of divergence ranking is in the first Q2 of the second ranking; the sum of divergence ranking is the sum of time divergence ranking and location divergence ranking, and the second ranking result is obtained by ranking the multiple characters in ascending order of the sum of divergence ranking, and Q2 is a positive integer.
11. The method according to any one of claims 3 to 10, It is characterized in that The identifying the first social circle according to the intimacy between each two characters and the character image heterogeneous graph includes: According to the intimacy between every two characters and the character image heterogeneous graph, identifying a first social circle and at least one second social circle, where the second social circle is different from the first social circle; The method further comprises: Determine the circle intimacy of the first social circle and each of the second social circles, wherein the circle intimacy is the average of the intimacy between each person in the social circle and the central person; Determine, in the first social circle and each of the second social circles, a social circle whose circle intimacy satisfies a second preset condition as an intimate social circle; A second photo album is generated based on the close social circle, wherein the second photo album includes images within a second circle, and the images within the second circle are images including at least one person in the close social circle.
12. The method according to claim 11, It is characterized in that The second preset condition includes: The circle intimacy is greater than a preset threshold, or the circle intimacy ranking is in the top X of the third ranking, and the third ranking is obtained by ranking the first social circle and each of the second social circles in descending order according to the circle intimacy, and X is a positive integer.
13. The method according to any one of claims 1 to 12, It is characterized in that Determining the intimacy between every two characters among the plurality of characters according to the character image heterogeneous graph includes: Determine the intimacy between a person corresponding to a non-co-occurrence image and other persons among the multiple persons as 0, wherein the non-co-occurrence image is an image including one person; Determine a co-occurrence image set from the multiple images according to the person image heterogeneous graph, wherein the co-occurrence image set includes multiple co-occurrence images, and the co-occurrence images are images including at least two persons; Calculating intimacy respectively according to the co-occurrence images of every two characters in the co-occurrence image set, to obtain the intimacy between every two characters included in the images in the co-occurrence image set; The intimacy calculation includes: Determine the mutual information between the second person and the third person according to the number of images corresponding to the second person in the co-occurrence image set, the number of images corresponding to the third person, the number of target co-occurrence images, and the total number of images in the co-occurrence image set; the second person and the third person are any two persons included in the images in the co-occurrence image set, and the target co-occurrence image is a co-occurrence image including the second person and the third person; The intimacy between the second person and the third person is determined based on the mutual information.
14. The method according to claim 13, It is characterized in that The person image heterogeneous graph includes shooting time and shooting location of the multiple images; and determining the intimacy between the second person and the third person according to the mutual information includes: According to the shooting time of the target co-occurrence image, statistically analyzing the distribution of the target co-occurrence image in multiple preset time intervals to obtain a second time distribution; Determine a divergence between the second time distribution and the uniform time distribution to obtain a time distribution divergence of the target co-occurrence image; According to the shooting location of the target co-occurrence image, statistics are collected on the distribution of the target co-occurrence image in a plurality of known locations to obtain a second location distribution, wherein the plurality of known locations include the shooting locations of all images in the gallery; Determining a divergence between the second location distribution and a uniform location distribution to obtain a location distribution divergence of the target co-occurrence image; The intimacy between the second person and the third person is determined according to the mutual information, the time distribution divergence of the target co-occurrence image, and the location distribution divergence of the target co-occurrence image.
15. The method according to claim 14, It is characterized in that The determining the intimacy between the second person and the third person according to the mutual information, the time distribution divergence of the target co-occurrence image, and the location distribution divergence of the target co-occurrence image includes: The intimacy between the second character and the third character is determined according to the following formula: closure(a,b)=PMI(a,b)×(1-D ab,t )×(1-D ab,l ); Wherein, closure(a,b) represents the intimacy between the second person and the third person, PMI(a,b) represents the mutual information, and D aa,t represents the temporal distribution divergence of the target co-occurrence image, D ab,l Represents the location distribution divergence of the target co-occurrence image.
16. The method according to any one of claims 1 to 15, It is characterized in that The character image isomorphic graph includes shooting time information and shooting locations of the multiple images, and the method further includes: Determining the type of the first social circle according to the character image heterogeneous graph, the type of the first social circle representing the type of social relationship between the characters in the first social circle and the central character, the time distribution divergence and the location distribution divergence of the image corresponding to the central character satisfy a first preset condition, the time distribution divergence is the distribution divergence of the shooting time of the image, and the location distribution divergence is the distribution divergence of the shooting location of the image; A theme of the first photo album is determined according to the type of the first social circle.
17. The method according to claim 16, It is characterized in that The determining the type of the first social circle according to the character image heterogeneous graph includes: Inputting the heterogeneous graph of character images into a graph neural network model, extracting a character feature vector of each of the multiple characters and an image feature vector of each of the multiple images; Inputting the character feature vector and the image feature vector into a multi-layer perceptron model to predict the social relationship type between each of the multiple characters and the central character; The type of the first social circle is determined according to the type of social relationship between the characters in the first social circle and the central character.
18. The method according to claim 17, It is characterized in that The determining the type of the first social circle according to the type of social relationship between the characters in the first social circle and the central character includes: Determine the number of people corresponding to each type of social relationship in the first social circle; The social relationship type corresponding to the largest number of persons is determined as the type of the first social circle.
19. The method according to claim 17 or 18, It is characterized in that The method further comprises: Modify the social relationship type of the person to be modified in the first social circle to the type of the first social circle; the person to be modified is a person in the first social circle, whose social relationship type with the central person is different from the type of the first social circle.
20. The method according to any one of claims 1 to 19, It is characterized in that The determining of a character image heterogeneous graph according to a plurality of images in a gallery application comprises: Generate a face clustering result according to the multiple images in the gallery application, the face clustering result represents the correspondence between the multiple images, the multiple faces and the multiple characters, wherein, among the multiple images, an image including a first face corresponds to the first face, and among the multiple characters, the character to which the first face belongs corresponds to the first face, and the first face is any one of the multiple faces; The character image heterogeneous graph is determined according to the face clustering result.
21. An electronic device, It is characterized in that include: Processors, memory, and interfaces; The processor, the memory and the interface cooperate with each other so that the electronic device executes the method as claimed in any one of claims 1 to 20.
22. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 20.
Citation Information
Patent Citations
Album creation method and apparatus
CN104991910A
Social relation analysis method and system based on multi-modal data and storage medium
CN115293920A
Method for automatic storytelling for photo albums using social network context
US20120014560A1
Automatic generation of people groups and image-based creations
US20220036614A1
Cited By
Media file display method and device
WO2026218692A1
