A face image classification method, intelligent terminal and storage medium

Abstract

The present application discloses a facial image classification method, an intelligent terminal, and a computer-readable storage medium, wherein the facial image classification method includes: obtaining multiple facial images; determining the similarity between an unclassified first facial image and the remaining facial images; in response to the existence of at least one image whose similarity to the first facial image is greater than or equal to a set first similarity threshold, classifying the first facial image and at least one image into one category of images; in response to the absence of an image whose similarity to the first facial image is greater than or equal to the first similarity threshold, determining a second facial image with the greatest similarity to the first facial image, and classifying the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image. In the above manner, the similarity between the unclassified facial image and the remaining facial images is used for classification, which can reduce the misclassification and misclassification of facial images.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to a face image classification method, an intelligent terminal, and a computer-readable storage medium. Background Art

[0002] Face classification technology is widely used. For example, it can be used in mobile phone albums to classify different characters, allowing users to better manage and view albums. In addition, if users need to create similar albums such as wonderful moments and wonderful collections, it can also make it easier for users to select pictures. In addition to mobile phone albums, it can also be used in page programs or applications with shared album functions. When users create new shared albums, it is more convenient for users to select pictures without having to search for them one by one, thereby optimizing the user experience with the help of technical means.

[0003] Existing face detection and recognition technologies are relatively mature, but there is no universal strategy for how to classify faces in an album. The more natural idea is to compare the pictures two by two. If the similarity between the two pictures is less than a given threshold, the two pictures are classified into one category. However, this classification method is not stable and is prone to misclassification and misclassification. For example, two photos of the same person may be quite different, so it is difficult to classify them into the same category; or two photos of different people may be quite similar and be mistakenly classified into the same category. Summary of the invention

[0004] To solve the above problems, the present application provides a facial image classification method, an intelligent terminal and a computer-readable storage medium, which can improve the stability of facial image classification and thereby reduce the misclassification and misclassification of facial images.

[0005] A technical solution adopted in the present application is: to provide a facial image classification method, the method comprising: obtaining multiple facial images; determining the similarity between an unclassified first facial image and the remaining facial images; in response to the existence of at least one image whose similarity to the first facial image is greater than or equal to a set first similarity threshold, classifying the first facial image and the at least one image into one category of images; in response to the absence of an image whose similarity to the first facial image is greater than or equal to the first similarity threshold, determining a second facial image with the greatest similarity to the first facial image, and classifying the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image.

[0006] Optionally, classifying the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image includes: in response to the similarity between the second facial image and the first facial image being greater than or equal to a set second similarity threshold, classifying the first facial image and / or the second facial image according to whether the second facial image has been classified; wherein the second similarity threshold is less than the first similarity threshold; in response to the similarity between the second facial image and the first facial image being less than the second similarity threshold, determining an image class with the greatest similarity to the second facial image among at least one classified image class, and classifying the first facial image and / or the second facial image according to the image class with the greatest similarity to the second facial image and the similarity between the second facial image.

[0007] Optionally, the first facial image and / or the second facial image is classified based on whether the second facial image has been classified, including: in response to the second facial image having been classified, attributing the first facial image to the image class to which the second facial image belongs; in response to the second facial image not being classified, determining an image class with the greatest similarity to the second facial image among at least one classified image class, and classifying the first facial image and / or the second facial image based on the image class with the greatest similarity to the second facial image and the similarity between the second facial images.

[0008] Optionally, the first facial image and / or the second facial image are classified according to the image class with the greatest similarity to the second facial image and the similarity of the second facial image, including: in response to the first image class having the greatest similarity to the second facial image and the similarity between the first image class and the second facial image being greater than or equal to a set third similarity threshold, the first facial image and the second facial image are classified into the first image class; in response to the first image class having the greatest similarity to the second facial image and the similarity between the first image class and the second facial image being less than a set third similarity threshold, the first facial image and the second facial image are classified into one class of images.

[0009] Optionally, the first facial image and / or the second facial image is classified according to the similarity between the second facial image and an image class with the greatest similarity, including: in response to the second facial image being unclassified and at least one image class not existing, classifying the first facial image and the second facial image into one class of images.

[0010] Optionally, the first facial image and / or the second facial image are classified according to the image class with the greatest similarity to the second facial image and the similarity of the second facial image, including: in response to the second image class having the greatest similarity to the second facial image and the similarity of the second facial image to the second image class being greater than or equal to a set fourth similarity threshold, the second facial image is classified into the second image class; in response to the second image class having the greatest similarity to the second facial image and the similarity of the second facial image to the second image class being less than a set fourth similarity threshold, the first facial image and the second facial image are each classified into one category of images.

[0011] Optionally, the first facial image and / or the second facial image is classified according to the image class with the greatest similarity to the second facial image and the similarity of the second facial image, including: in response to the absence of at least one image class, classifying the first facial image and the second facial image into one class of images respectively.

[0012] Optionally, the method also includes: obtaining a third facial image; determining the similarity between the third facial image and all image classes; in response to the third image class having the greatest similarity to the third facial image and the similarity between the third image class and the third facial image being greater than or equal to a set fifth similarity threshold, classifying the third facial image into a third image class; in response to the third image class having the greatest similarity to the third facial image and the similarity between the third image class and the third facial image being less than a set fifth similarity threshold, classifying the third facial image into one class of images.

[0013] Optionally, determining the similarity between the unclassified first facial image and the remaining facial images includes: performing image recognition on multiple facial images to obtain the number of faces and feature vectors of the multiple facial images; removing facial images with a number of faces greater than 1; determining the feature vector of the unclassified first facial image and the distance between it and the feature vectors of the remaining facial images to determine the similarity between the unclassified first facial image and the remaining facial images.

[0014] Optionally, the method also includes: after any face image is classified into an image class or any face image is attributed to an image class, updating the average feature vector of the corresponding image class; wherein the average feature vector is the average value of the feature vectors of all face images in the image class, and the average feature vector is used to determine the similarity between the image class and the face image.

[0015] Another technical solution adopted in the present application is: to provide an intelligent terminal, which includes a processor and a memory connected to the processor; wherein program data is stored in the memory, and the processor calls the program data stored in the memory to execute the face image classification method as described above.

[0016] Another technical solution adopted by the present application is: providing a computer-readable storage medium, in which program data is stored. When the program data is executed by a processor, it is used to implement the facial image classification method as described above.

[0017] The facial image classification method provided by the present application includes: obtaining multiple facial images; determining the similarity between an unclassified first facial image and the remaining facial images; in response to the existence of at least one image whose similarity to the first facial image is greater than or equal to a set first similarity threshold, classifying the first facial image and the at least one image into one category of images; in response to the absence of an image whose similarity to the first facial image is greater than or equal to the set first similarity threshold, determining a second facial image with the greatest similarity to the first facial image, and classifying the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image. In the above manner, different classifications are performed according to the similarity between the unclassified facial image and the remaining facial images, which can improve the stability of facial image classification and thereby reduce the misclassification and misclassification of facial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0019] Figure 1 It is a flowchart of the first embodiment of the face image classification method provided by the present application;

[0020] Figure 2 is a flow chart of an embodiment of determining the similarity between an unclassified first face image and remaining face images;

[0021] Figure 3 is a schematic diagram of a flow chart of an embodiment of performing image recognition on multiple face images;

[0022] Figure 4 is a flow chart of an embodiment of step 123;

[0023] Figure 5 It is a flowchart of the second embodiment of the face image classification method provided by the present application;

[0024] Figure 6 It is a flowchart of the third embodiment of the face image classification method provided by the present application;

[0025] Figure 7 A schematic diagram of the structure of a smart terminal provided for this application;

[0026] Figure 8 A schematic diagram of the structure of an embodiment of a computer-readable storage medium provided in this application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0028] Reference to "embodiments" in an application means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0029] The steps in the embodiments of the present application do not necessarily have to be processed in the described order of steps. The steps can be selectively rearranged, or the steps in the embodiments can be deleted, or the steps in the embodiments can be added as needed. The step descriptions in the embodiments of the present application are only optional sequence combinations and do not represent all step sequence combinations in the embodiments of the present application. The order of steps in the embodiments cannot be considered as a limitation of the present application.

[0030] The term "and / or" in the embodiments of the present application refers to any and all possible combinations of one or more of the associated enumerated items. It should also be noted that when used in this specification, "include / comprise" specifies the presence of stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components and / or their groups.

[0031] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0032] In addition, although the terms "first", "second", etc. are used many times in this application to describe various elements (or various thresholds or various applications or various instructions or various operations), etc., these elements (or thresholds or applications or instructions or operations) should not be limited by these terms. These terms are only used to distinguish one element (or threshold or application or instruction or operation) from another element (or threshold or application or instruction or operation). For example, a first facial image can be called a second facial image, and a second facial image can also be called a first facial image without departing from the scope of this application. The first facial image and the second facial image are both facial images, but they are not the same facial images.

[0033] The intelligent terminal (such as a mobile terminal) of the embodiment of the present application can be implemented in various forms. Among them, the intelligent terminal can be a mobile terminal including a collection and identification device (such as a camera and a video recorder), a mobile phone, a smart phone, a laptop, a personal digital assistant (PDA, Personal Digital Assistant), a tablet computer (PAD), etc. that can store image information and be accessed or send image information. The intelligent terminal can also be a fixed terminal that can store image information and be accessed or send image information such as a digital broadcast transmitter, a digital TV, a desktop computer, etc. Below, it is assumed that the terminal is a mobile terminal. However, it will be understood by those skilled in the art that, in addition to components specifically used for mobile purposes, the construction according to the embodiment of the present application can also be applied to fixed type terminals.

[0034] In the prior art, there are many problems in comparing pictures pair by pair and then classifying them. For example, two pictures of the same person may be quite different, so it is difficult to classify them into the same category; or two pictures of different people may be quite similar and be mistakenly classified into the same category.

[0035] Therefore, this application designs a set of classification logic for the classification of single faces in face images. In the comparison of face images, the face library is continuously established and enriched, so that a more representative average face can be obtained, and thus the situations of misclassification and wrong classification can be alleviated.

[0036] See also Figure 1 , Figure 1 : is a flowchart of a first embodiment of a face image classification method provided by the present application, the method comprising:

[0037] Step 11: Obtain multiple face images.

[0038] Specifically, an image library in an intelligent terminal is accessed, and a plurality of original face images in the image library are extracted. Alternatively, a plurality of face images may be obtained by receiving image information sent by other intelligent terminals or application terminals (such as a server).

[0039] In one embodiment, the image system of the smartphone is accessed and multiple facial images are obtained, wherein the multiple facial images may be all images in the image system of the smartphone, or a part of specific images, or multiple facial images may be obtained at a predetermined time, which is not specifically limited here.

[0040] Step 12: Determine the similarity between the unclassified first face image and the remaining face images.

[0041] After obtaining multiple face images, the unclassified face images among all face images are cyclically classified until all face images are classified into one image category, and the classified face images will be marked with a "classified" logo and will no longer participate in the cyclic classification process of the remaining face images. That is, after the cyclic classification process is completed, all the acquired face images will be classified into an existing image category or become a new image category.

[0042] Furthermore, at the start of each cyclic classification, it is necessary to determine the similarity between an unclassified target face image and the remaining face images. The remaining face images refer to the remaining face images other than the target face image among all the acquired face images. Optionally, after acquiring multiple face images, all face images may be compared in pairs to obtain the similarity between all face images, and all these similarities may be used as a similarity set to call the similarity between the corresponding face images in each cyclic classification process.

[0043] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of determining the similarity between the unclassified first face image and the remaining face images, wherein step 12 specifically includes the following steps:

[0044] Step 121: performing image recognition on multiple face images to obtain the number of faces and feature vectors of the multiple face images.

[0045] Specifically, by calling a face recognition interface, the acquired face image is sent to a face recognition system of a smart terminal or server to recognize the face image and obtain the image information of the face image. The image information of the face image includes the attribute information of the face image and the number of faces. The attribute information is the parameter information of the face image, such as the acquisition time, shooting time, location, image size, etc.

[0046] Furthermore, by accessing the image system of the smart terminal, the multiple facial images obtained include not only the user's face, but also other parts of the user and the surrounding environment. In addition, the number of faces in the facial image may be only one or more. Therefore, in order to improve the recognition speed and classification efficiency of facial images, a convolutional neural network is used in the facial recognition system to recognize facial images, obtain the number of faces in each facial image, and mark and determine the facial area of ​​the facial image. According to the determined and marked facial area of ​​the facial image, the facial image is feature extracted using an application algorithm program to obtain a feature vector of the facial image.

[0047] Optionally, in one embodiment, the face recognition system performs sample training on the received face images to obtain multiple face recognition models, and uses the face recognition models to perform feature extraction on the face images to obtain feature vectors of the face images.

[0048] Among them, the above-mentioned face recognition model can be an artificial neural network model, a support vector machine and other non-neural network models. For example, it can be a convolutional neural network model. Convolutional Neural Network (CNN) is a kind of deep artificial neural network. Usually, a convolutional neural network can include multiple feature extraction layers (also called convolution layers) and multiple downsampling layers (also called pooling layers). Among them, the feature extraction layer and the downsampling layer are alternately connected. Each feature extraction layer can include at least one convolution kernel. For a feature extraction layer, a convolution kernel of the layer is used to convolve with the output of the previous layer to obtain a feature map. The downsampling layer is used to calculate the local average and dimension reduction of the convolution result of the output of the feature extraction layer connected to it. Among them, the convolution kernel in the feature extraction layer includes multiple weights. The weights in the convolution kernel can be obtained by training multiple samples. Each convolution kernel of the convolutional neural network uses local weight sharing when extracting feature maps from the image, which can reduce the complexity of the neural network model.

[0049] See also Figure 3 , Figure 3 The flowchart of an embodiment of performing image recognition on multiple face images to obtain feature vectors of multiple face images is shown in step 121. Step 121 specifically includes the following steps:

[0050] Step 1211: Extract Haar features of the face image.

[0051] Among them, Haar feature is a kind of image feature. Since the same kind of objects have the same Haar feature, different kinds of objects have different Haar features. For example, face images have the same Haar feature, and face images and non-face images have different Haar features. Therefore, Haar feature can be used for object recognition.

[0052] In order to improve the accuracy of determining the facial area of ​​a facial image, the method provided in an embodiment of the present application can use windows of different sizes and positions as search windows to search the facial image when extracting the Haar features of the facial image, and extract the Haar features within each search window.

[0053] Step 1212: Determine the face area of ​​the face image.

[0054] If the extracted Haar feature of the face image is a Haar feature corresponding to the face part, the convolutional neural network model determines the location area of ​​the Haar feature of the face image as the face area. After completing the search of the face image through the above method, the convolutional neural network model merges all the face areas to obtain the face area of ​​the face image.

[0055] Step 1213: Locate the facial key points in the face area.

[0056] Among them, the key points of the face refer to the key points in the face area that represent the features of the facial features, including the key points for locating eyebrows, the key points for locating eyes, the key points for locating noses, the key points for locating mouths, and the key points for locating facial contours. The number of key points of the face can be 80, 100, etc.

[0057] Step 1214: normalize the located face image to obtain a feature vector of the face image.

[0058] In order to make the facial images taken under different imaging conditions such as lighting intensity, direction, distance, posture, etc. consistent and to reduce the amount of calculation when recognizing multiple facial images, the convolutional neural network model performs normalization processing such as translation, rotation, scaling, and standard cutting on the located facial images, and calculates the feature vector of the normalized facial image through the convolutional neural network algorithm.

[0059] Step 122: remove the facial images with more than 1 face.

[0060] Optionally, face images with more than 1 face are classified into one category of images, and face images without faces are classified into another category of images. That is, the image category with more than 1 face and the image category without faces do not participate in the subsequent face image classification method.

[0061] Step 123: Determine the distance between the feature vector of the unclassified first facial image and the feature vectors of the remaining facial images to determine the similarity between the unclassified first facial image and the remaining facial images.

[0062] See also Figure 4 , Figure 4It is a schematic flowchart of an embodiment of step 123, and step 123 specifically includes the following steps:

[0063] Step 1231: Determine the feature vector of the unclassified first face image and the distances from the feature vectors of the remaining face images.

[0064] Specifically, the distances between the feature vectors of the unclassified first face image and the remaining face images can be determined by Euclidean distance or cosine distance.

[0065] Taking the determination of the distance between the feature vectors of two face images by cosine distance as an example:

[0066] When the image processing system calculates the distances between the feature vector of the unclassified first face image and the feature vectors of the remaining face images, it can calculate the cosine distance between the two feature vectors. For example, for the first face image represented by A and a certain face image in the remaining face images represented by B. Among them, the face feature vector of face image A is (a1, a2,..., a200), and the face feature vector of face image B is (b1, b2,..., b200), then the cosine distance between the face feature vector of face image A and the face feature vector of face image B is:

[0067]

[0068] Among them, when the left side of the formula is equal to the right side, it means that the cosine distance between the face feature vector of face image A and the face feature vector of face image B is the smallest.

[0069] Step 1232: Determine the similarity between the unclassified first face image and the remaining face images.

[0070] Specifically, perform a relationship conversion on the Euclidean distance or cosine distance between the feature vector of the unclassified first face image and the feature vectors of the remaining face images to determine the similarity between the unclassified first face image and the remaining face images.

[0071] Taking the determination of the similarity between the unclassified first face image and the remaining face images by cosine distance as an example:

[0072] The cosine distance between the face feature vector of the first face image A and the face feature vector of the face image B in the remaining face images is equal to Cos<A. That is to say, the similarity Cos between the face features of the first face image A and the face features of the face image B can be expressed as:

[0073]

[0074] Among them, the similarity Cos is less than or equal to 1 and greater than or equal to 0.

[0075] It can be understood that the greater the distance between the feature vectors of the unclassified first face image and the remaining face images, the smaller the similarity between the unclassified first face image and the remaining face images; the smaller the distance between the feature vectors of the unclassified first face image and the remaining face images, the greater the similarity between the unclassified first face image and the remaining face images. The similarity between the unclassified first face image and the remaining face images is any value between 0 and 1.

[0076] Furthermore, the distance between the facial feature vector of the first facial image A and the facial feature vector of each facial image in the remaining facial images is continued to be calculated, and then the distance between the facial feature vector of the first facial image A and the facial feature vector of each facial image in the remaining facial images is converted to obtain the corresponding similarity, and saved as a similarity set for subsequent facial image classification.

[0077] Step 13: In response to the existence of at least one image having a similarity with the first facial image greater than or equal to a set first similarity threshold, the first facial image and the at least one image are classified into one type of image.

[0078] Optionally, the first similarity threshold can be any threshold between 0 and 1. Taking the first similarity threshold of 0.5 as an example, the system sets the facial similarity to be greater than or equal to 50%, that is, the two faces are the same face. In response to the existence of an image in the remaining facial images whose similarity to the first facial image is greater than or equal to 0.5, it is considered that the face corresponding to the first facial image is greater than or equal to 50% similar to the face corresponding to the image, then the two facial images corresponding to the same face are classified into one category of images, indicating that this is an image category of the same person. In response to the existence of an image in the remaining facial images whose similarity to the first facial image is greater than or equal to 0.5, proceed to step 14.

[0079] Step 14: In response to the absence of an image whose similarity to the first facial image is greater than or equal to a first similarity threshold, determine a second facial image with the greatest similarity to the first facial image, and classify the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image.

[0080] Specifically, the second facial image is an image among the remaining facial images whose similarity with the first facial image is less than the first similarity threshold and is closest to the first similarity threshold. Among them, the second facial image can be regarded as one facial image or as multiple facial images in different situations. For example, there may be only one facial image or multiple facial images among the remaining facial images whose similarity with the first facial image is less than the first similarity threshold. In the case where there are multiple such facial images, after the second facial image closest to the first similarity threshold is classified, the second facial image closest to the first similarity threshold becomes the new second facial image closest to the first similarity threshold, and the classification is terminated after all images whose similarity with the first facial image is less than the first similarity threshold and are closest to the first similarity threshold are completed.

[0081] Furthermore, step 14 specifically includes the following steps:

[0082] Step 141: In response to the similarity between the second facial image and the first facial image being greater than or equal to a set second similarity threshold, classifying the first facial image and / or the second facial image according to whether the second facial image has been classified; wherein the second similarity threshold is less than the first similarity threshold.

[0083] Specifically, the purpose of setting the second similarity threshold is to prevent a second facial image whose similarity to the first facial image is too small from being classified into the same category as the first facial image. For example, the similarity between a second facial image and the first facial image is 0.1, and 0.1 is less than the set first similarity threshold of 0.5. At this time, the second facial image and the first facial image cannot be considered to be images of the same face, and a second similarity threshold (such as 0.25, 0.3, etc.) that is less than the first similarity threshold of 0.5 is set. If the similarity between the second facial image and the first facial image is less than the set first similarity threshold and greater than or equal to the set second similarity threshold, then the second facial image and the first facial image are considered to be images of the same face.

[0084] Further, in response to the existence of a second facial image whose similarity to the first facial image is less than the set first similarity threshold and greater than or equal to the set second similarity threshold, the image has been classified in the previous image classification process, and the first facial image and / or the second facial image are classified according to the classification of the second facial image. In response to the absence of a second facial image whose similarity to the first facial image is less than the set first similarity threshold and greater than or equal to the set second similarity threshold, the process proceeds to step 142.

[0085] For example, image 1 is the first face image, image 2 is the second face image, and image 3 is the third face image. The similarity between image 1 and image 2 is greater than the first similarity threshold, the similarity between image 1 and image 3 is less than the first similarity threshold and greater than the second similarity threshold, and the similarity between image 2 and image 3 is less than the first similarity threshold and greater than the second similarity threshold. In the previous image classification process, image 1 and image 2 have been classified into the same category of images. Since the similarity between image 2 and image 3 meets the condition of being less than the first similarity threshold and greater than the second similarity threshold, it is considered that image 2 and image 3 are also images of the same face. At this time, image 3 is classified according to the classification of image 2.

[0086] Furthermore, step 141 specifically includes the following steps:

[0087] Step 1411: In response to the second facial image being classified, assigning the first facial image to the image class to which the second facial image belongs.

[0088] If the second facial image is not classified, the process proceeds to step 1412 .

[0089] Step 1412: In response to the second facial image being unclassified, determine an image class among at least one classified image class that has the greatest similarity to the second facial image, and classify the first facial image and / or the second facial image based on the image class that has the greatest similarity to the second facial image and the similarity between the second facial images.

[0090] Specifically, based on the image class with the greatest similarity to the second facial image and the similarity of the second facial image, it is determined whether the facial image in the image class with the greatest similarity to the second facial image and the second facial image are images of the same facial image, and then the first facial image and / or the second facial image are classified.

[0091] Specifically, step 1412 includes the following steps:

[0092] Step a1: In response to the first image class and the second facial image having the greatest similarity, and the similarity between the first image class and the second facial image being greater than or equal to a set third similarity threshold, the first facial image and the second facial image are classified into the first image class.

[0093] Optionally, the third similarity threshold is less than or equal to the first similarity threshold, and there is no specific limit on the magnitude between the third similarity threshold and the second similarity threshold. For example, if the first similarity threshold is 0.5 and the second similarity threshold is 0.25, the third similarity threshold may be 0.25, 0.3 or 0.2.

[0094] If the similarity between the first image class and the second face image is less than the set third similarity threshold, it is considered that the face image in the first image class and the second face image do not represent the same face, and step a2 is entered.

[0095] Step a2: In response to the first image category and the second facial image having the greatest similarity and the first image category and the second facial image having a similarity less than a set third similarity threshold, the first facial image and the second facial image are classified into one category.

[0096] It is understandable that, in response to the similarity between the second face image and the first face image being greater than or equal to the set second similarity threshold and the second face image being unclassified, the first face image and / or the second face image are classified according to the image class with the greatest similarity to the second face image and the similarity of the second face image. There is a situation that cannot be ignored, in response to the second face image being unclassified and at least one image class not existing. That is, there is no image that has been classified yet, and this is the first classification in the cyclic classification process, and there is no image class yet. Since this situation is met, the first face image and the second face image are classified into one class of images.

[0097] Step 142: In response to the similarity between the second facial image and the first facial image being less than a second similarity threshold, determine an image class among at least one classified image class that has the greatest similarity to the second facial image, and classify the first facial image and / or the second facial image based on the image class that has the greatest similarity to the second facial image and the similarity between the second facial images.

[0098] Specifically, in response to the similarity between the second facial image and the first facial image being less than the second similarity threshold, it is considered that the second facial image and the first facial image do not represent images of the same face, and the second facial image and the first facial image need to be classified separately.

[0099] Furthermore, step 142 specifically includes the following steps:

[0100] Step 1421: In response to the second image class having the greatest similarity to the second facial image and the second facial image having a similarity greater than or equal to a set fourth similarity threshold, the second facial image is classified into the second image class.

[0101] Optionally, the fourth similarity threshold is less than or equal to the first similarity threshold, and there is no specific limitation on the magnitude between the fourth similarity threshold and the second similarity threshold and the third similarity threshold. For example, if the first similarity threshold is 0.5, the second similarity threshold is 0.25, and the third similarity threshold is 0.25, the fourth similarity threshold may be 0.25, 0.3, or 0.2.

[0102] If the similarity between the second image class and the second facial image is less than the set fourth similarity threshold, it is considered that the facial image in the second image class and the second facial image do not represent the same face, and step 1422 is entered.

[0103] Step 1422: In response to the second image category and the second facial image having the greatest similarity and the second facial image having a similarity with the second image category being less than a set fourth similarity threshold, the first facial image and the second facial image are each classified into one category of images.

[0104] It is understandable that, in response to the similarity between the second face image and the first face image being less than the second similarity threshold, the first face image and / or the second face image are classified according to the image class with the greatest similarity to the second face image and the similarity of the second face image. There is a situation that cannot be ignored, in response to the absence of at least one image class. That is, there is no image that has been classified. At this time, it is the first classification in the cyclic classification process, and there is no image class. Since this situation is met, the first face image and the second face image are each classified into a class of images.

[0105] The smart terminal in the embodiment of the present application is implemented in various forms. Among them, the image library in the smart terminal (such as a camera and a mobile phone) may frequently update images, that is, frequently add new images to the image library, or delete images from the image library. Therefore, in another embodiment, it also includes a method for obtaining the newly added image from the image library to perform face image classification.

[0106] See also Figure 5 , Figure 5 : is a flow chart of a second embodiment of a face image classification method provided by the present application, the method comprising:

[0107] Step 21: Obtain a third face image.

[0108] Specifically, an image library in an intelligent terminal is accessed, and a newly added (or newly stored) face image in the image library is extracted.

[0109] Step 22: Determine the similarity between the third face image and all image classes.

[0110] The method of obtaining the third facial image and determining the similarity between the third facial image and all image categories is similar to steps 11 and 12 in the above embodiment and will not be described in detail here.

[0111] Step 23: In response to the third image class having the greatest similarity to the third facial image and the similarity between the third image class and the third facial image being greater than or equal to a set fifth similarity threshold, the third facial image is classified into the third image class.

[0112] Optionally, the fifth similarity threshold is less than or equal to the first similarity threshold, and there is no specific limitation on the magnitude between the fifth similarity threshold and the second similarity threshold, the third similarity threshold, and the fourth similarity threshold. For example, if the first similarity threshold is 0.5, the second similarity threshold is 0.25, the third similarity threshold is 0.25, and the fourth similarity threshold is 0.25, the fifth similarity threshold may be 0.25, 0.3, or 0.2.

[0113] If the similarity between the third image class and the third face image is less than the set fifth similarity threshold, it is considered that the face image in the third image class and the third face image do not represent the same face, and step 24 is entered.

[0114] Step 24: In response to the third image class having the greatest similarity to the third facial image and the third image class having the greatest similarity to the third facial image and the similarity between the third image class and the third facial image being less than a set fifth similarity threshold, the third facial image is classified into one category of images.

[0115] Optionally, after any face image is classified into an image class or any face image is attributed to an image class, the average feature vector of the corresponding image class is updated, wherein the average feature vector is the average value of the feature vectors of all face images in the image class (i.e., the average face vector), and the average feature vector is used to determine the similarity between the image class and the face image.

[0116] Among them, the face library is continuously established and enriched in the similarity comparison of face images and / or image classes, so that a more representative average face vector can be obtained. The distance calculated from the average face vector will be more stable than the distance calculated from the face of a single face image, so the situations of misclassification and wrong classification can be alleviated.

[0117] Optionally, after the average feature vector of the corresponding image class is updated, the image class is added to a face library for storage, so as to facilitate subsequent updating of the image class or calling of a certain image class.

[0118] See also Figure 6 , Figure 6 : is a flowchart of a third embodiment of a face image classification method provided by the present application, the method comprising:

[0119] (1) Input a picture, call the face recognition interface to recognize the picture, and return the number of faces in the picture and the facial feature vector F of each person.

[0120] (2) Determine whether there is a picture among all the pictures that satisfies the number of faces n = 1. If not, no classification is performed; if so, each picture is classified cyclically, with a total of N pictures, starting from the first picture.

[0121] (3) Perform traversal and loop classification on the i-th picture, where i=x+1.

[0122] (4) Calculate the similarity T between all remaining pictures and the face in picture i, and record them in an array A. Calculate the number of pictures Na in array A that satisfy a condition greater than the threshold T1. Determine whether Na>0 is satisfied. If so, classify all pictures that meet the condition and picture i into one category and put them into the face database, and update the average face vector F of this category. If not, determine whether there is a similarity T0 of picture j in the remaining pictures that satisfies T2<T0<T1.

[0123] (4-1) If there is no image number Na>0 in array A that satisfies the threshold value T1, and there is an image j in the remaining images whose similarity T0 satisfies T2<T0<T1. Then determine whether image j has been classified. If image j has been classified, classify image i into the category to which image j belongs, put it into the face database, and update the average face vector F of this category. If image j has not been classified, determine whether the face database exists. If the face database does not exist, classify image j and image i into one category, put them into the face database, and update the average face vector F of this category. If the face database exists, calculate the similarity between image j and each category in the face database, and obtain the minimum similarity value Tx. Then determine whether T3<Tx is satisfied. If T3<Tx is satisfied, classify image j into the category corresponding to the similarity Tx, and update the average face vector F of this category. If T3<Tx is not satisfied, determine whether i≤N is satisfied. If i≤N is satisfied, return to step (3) and enter the next classification cycle.

[0124] (4-2) If there is no image in array A that satisfies the number Na>0 greater than the threshold T1, and there is no image j in the remaining images whose similarity T0 satisfies T2<T0<T1. Then determine whether the face database exists. If the face database does not exist, then determine whether i≤N is satisfied. If i≤N is satisfied, then return to step (3) and enter the next classification cycle. If the face database exists, then calculate the similarity between image j and each category in the face database, and obtain the minimum similarity value Ty. Then determine whether Ty>T4 is satisfied. If Ty>T4 is satisfied, then classify image j into the category corresponding to the similarity Ty, and update the average face vector F of the category. If Ty>T4 is not satisfied, then determine whether i≤N is satisfied. If i≤N is satisfied, then return to step (3) and enter the next classification cycle.

[0125] (5) In step (4-1) or step (4-2), if i≤N is not satisfied, the face database is merged, the unclassified images are classified as a separate category, and the classification is terminated.

[0126] Different from the prior art, the face image classification method provided in this embodiment includes: obtaining multiple face images; determining the similarity between an unclassified first face image and the remaining face images; in response to the existence of at least one image whose similarity to the first face image is greater than or equal to a set first similarity threshold, classifying the first face image and the at least one image into a category of images; in response to the absence of an image whose similarity to the first face image is greater than or equal to a set first similarity threshold, determining a second face image with the greatest similarity to the first face image, and classifying the first face image and / or the second face image according to the similarity between the first face image and the second face image. Through the above method, on the one hand, the unclassified face image and the remaining face images are cyclically classified according to the similarity between the two, the classification of the remaining face images, the similarity between the remaining face images and the image category with the greatest similarity, and whether there is an image category, so as to perform different classification strategies, which can improve the stability of face image classification. On the other hand, after any face image is classified into an image class or any face image is attributed to an image class, the average feature vector of the corresponding image class is updated, thereby reducing the misclassification and wrong classification of face images and optimizing the user experience.

[0127] See also Figure 7 , Figure 7 A structural diagram of a smart terminal provided in the present application, the smart terminal 100 includes a processor 101 and a memory 102 connected to the processor 101, wherein program data is stored in the memory 102, and the processor 101 calls the program data stored in the memory 102 to execute the above-mentioned face image classification method.

[0128] Optionally, in one embodiment, processor 101 is used to execute program data to implement the following method: obtain multiple facial images; determine the similarity between an unclassified first facial image and the remaining facial images; in response to the presence of at least one image whose similarity to the first facial image is greater than or equal to a set first similarity threshold, classify the first facial image and the at least one image into one category of images; in response to the absence of an image whose similarity to the first facial image is greater than or equal to the set first similarity threshold, determine a second facial image with the greatest similarity to the first facial image, and classify the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image.

[0129] The processor 101 may also be referred to as a CPU (Central Processing Unit). The processor 101 may be an electronic chip having the ability to process signals. The processor 101 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0130] The memory 102 can be a memory stick, a TF card, etc., which can store all the information in the smart terminal 100, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory 102. It stores and retrieves information according to the location specified by the processor 101. With the memory 102, the smart terminal 100 has a memory function and can ensure normal operation. The memory 102 of the smart terminal 100 can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose, and there is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but it is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the implementation of the intelligent terminal 100 described above is only illustrative. For example, the unclassified face images and the remaining face images are classified according to their similarity, the classification of the remaining face images, the similarity of the remaining face images with the image class with the greatest similarity, and whether there is an image class, so as to perform different classification strategies. This is only a collection method. There may be other division methods in actual implementation. For example, the unclassified face images and the remaining face images can be combined or can be aggregated into another system, or some features can be ignored or not executed.

[0132] In addition, each functional unit (such as a face library and an image library, etc.) in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0133] See also Figure 8 , Figure 8This is a schematic diagram of the structure of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 110 stores program instructions 111 that can implement all the above methods.

[0134] If the integrated units of the functional units in the various embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium 110. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer-readable storage medium 110 includes several instructions in a program instruction 111 to enable a computer device (which can be a personal computer, a system server, or a network device, etc.), an electronic device (such as MP3, MP4, etc., or a mobile terminal such as a mobile phone, a tablet computer, a wearable device, or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of various implementation methods of the present application.

[0135] Optionally, in one embodiment, when the program instruction 111 is executed by the processor, it is used to implement the following method: obtain multiple facial images; determine the similarity between an unclassified first facial image and the remaining facial images; in response to the existence of at least one image whose similarity to the first facial image is greater than or equal to a set first similarity threshold, classify the first facial image and the at least one image into one category of images; in response to the absence of an image whose similarity to the first facial image is greater than or equal to the set first similarity threshold, determine a second facial image with the greatest similarity to the first facial image, and classify the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media 110 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by a computer-readable storage medium 110. These computer-readable storage media 110 can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the program instructions 111 executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer-readable storage media 110 may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the program instructions 111 stored in the computer-readable storage medium 110 produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer-readable storage media 110 can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing program instructions 111 executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] In one embodiment, these programmable data processing devices include a processor and a memory. The processor may also be referred to as a CPU (Central Processing Unit). The processor may be an electronic chip having signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0141] The memory can be a memory stick, TF card, etc. It stores and retrieves information according to the location specified by the processor. According to the purpose, the memory can be divided into primary memory (internal memory) and auxiliary memory (external memory). There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but it is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0142] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made according to the description and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A face image classification method, It is characterized in that The method comprises: Acquire multiple face images; Determining the similarity between the unclassified first face image and the remaining face images; In response to the existence of at least one image having a similarity with the first facial image greater than or equal to a set first similarity threshold, classifying the first facial image and the at least one image into one type of image; In response to the absence of an image having a similarity with the first facial image greater than or equal to the first similarity threshold, determining a second facial image having the greatest similarity with the first facial image, and classifying the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image; The classifying the first facial image and / or the second facial image according to the similarity between the first facial image and the second facial image includes: In response to the similarity between the second facial image and the first facial image being greater than or equal to a set second similarity threshold, classifying the first facial image and / or the second facial image according to whether the second facial image has been classified; wherein the second similarity threshold is less than the first similarity threshold; In response to the similarity between the second facial image and the first facial image being less than the second similarity threshold, determine an image class among at least one classified image class that has the greatest similarity to the second facial image, and classify the first facial image and / or the second facial image based on the image class that has the greatest similarity to the second facial image and the similarity to the second facial image.

2. The method for classifying facial images according to claim 1, It is characterized in that The classifying the first facial image and / or the second facial image according to whether the second facial image has been classified includes: In response to the second facial image being classified, assigning the first facial image to the image class to which the second facial image belongs; In response to the second facial image being unclassified, determine an image class among at least one classified image class that has the greatest similarity to the second facial image, and classify the first facial image and / or the second facial image based on the image class that has the greatest similarity to the second facial image and the similarity to the second facial image.

3. The face image classification method according to claim 2, It is characterized in that The classifying the first facial image and / or the second facial image according to the image class having the greatest similarity to the second facial image and the similarity to the second facial image comprises: In response to the first image class and the second facial image having the greatest similarity, and the first image class and the second facial image having a similarity greater than or equal to a set third similarity threshold, classifying the first facial image and the second facial image into the first image class; In response to the first image category and the second facial image having the greatest similarity, and the first image category and the second facial image having the greatest similarity, and the similarity being less than the third similarity threshold, the first facial image and the second facial image are classified into one category.

4. The method for classifying facial images according to claim 1, It is characterized in that The classifying the first face image and / or the second face image according to the similarity between the second face image and the image class with the greatest similarity includes: In response to the second facial image being unclassified and not existing in at least one image category, the first facial image and the second facial image are classified into one category of images.

5. The method for classifying facial images according to claim 1, It is characterized in that The classifying the first facial image and / or the second facial image according to the image class having the greatest similarity to the second facial image and the similarity to the second facial image comprises: In response to the second image class having the greatest similarity to the second facial image and the second facial image having a similarity greater than or equal to a set fourth similarity threshold, classifying the second facial image into the second image class; In response to the second image category having the greatest similarity to the second facial image and the second facial image having the greatest similarity to the second image category and the second facial image having the second similarity to the second image category being less than the fourth similarity threshold, the first facial image and the second facial image are each classified into one category of images.

6. The method for classifying facial images according to claim 1, It is characterized in that The classifying the first facial image and / or the second facial image according to the image class having the greatest similarity to the second facial image and the similarity to the second facial image comprises: In response to the absence of at least one image class, the first face image and the second face image are each classified into one class of images.

7. The method for classifying facial images according to claim 1, It is characterized in that The method further comprises: Obtaining a third person's face image; Determining the similarity between the third face image and all image classes; In response to the third image class having the greatest similarity to the third facial image and the similarity between the third image class and the third facial image being greater than or equal to a set fifth similarity threshold, classifying the third facial image into the third image class; In response to the third image category having the greatest similarity to the third facial image and the similarity between the third image category and the third facial image being less than the set fifth similarity threshold, the third facial image is classified into one category of images.

8. The method for classifying facial images according to claim 1, It is characterized in that The determining of the similarity between the unclassified first face image and the remaining face images includes: Performing image recognition on the plurality of face images to obtain the number of faces and feature vectors of the plurality of face images; Remove the facial images with more than 1 face; Determine the distance between the feature vector of the unclassified first facial image and the feature vectors of the remaining facial images to determine the similarity between the unclassified first facial image and the remaining facial images.

9. The method for classifying facial images according to claim 8, It is characterized in that The method further comprises: After any face image is classified into an image class or any face image is attributed to an image class, the average feature vector corresponding to the image class is updated; wherein the average feature vector is the average value of the feature vectors of all face images in the image class, and the average feature vector is used to determine the similarity between the image class and the face image.

10. A smart terminal, It is characterized in that The intelligent terminal includes a processor and a memory connected to the processor, wherein the memory stores program data, and the processor calls the program data stored in the memory to execute the face image classification method as described in any one of claims 1-9.

11. A computer-readable storage medium having program instructions stored therein. It is characterized in that The program instructions are executed to implement the face image classification method as described in any one of claims 1-9.

Images