Face multi-attribute detection model establishment method and device

Abstract

The embodiment of the invention provides a face multi-attribute detection model establishment method and device. The method comprises the following steps: inputting a face sample image into a feature coding module of a to-be-trained face multi-attribute detection model, and inputting obtained coding features into a position index map acquisition module of the to-be-trained face multi-attribute detection model to obtain a predicted position index map of the face sample image; according to the position of each feature point included in each attribute in the predicted position index map of the face sample image on the face sample image, searching the coding feature of each attribute in the coding features; inputting the coding feature of each attribute into a classification module of a to-be-trained face multi-attribute detection model to obtain a prediction category of each attribute; and calculating a loss function, and updating neural network parameters of the feature coding module, the position index map acquisition module and the classification module by adopting the loss function value until the face multi-attribute detection model converges. According to the embodiment of the invention, the face multi-attribute detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for establishing a multi-attribute face detection model. Background Art

[0002] Currently, face attribute detection usually involves sending face images to a convolutional neural network for feature extraction, and then inputting the extracted features into a fully connected network to complete the classification of face attributes. This method has the following problems:

[0003] First, the whole face feature is not necessarily suitable for detecting a single attribute (such as eye shape, mouth shape, nose shape, hairstyle, whether wearing glasses, whether wearing a mask, etc.), because: when the whole face feature is used to detect a certain attribute, the features of the face area that are not related to the attribute will cause interference;

[0004] 2. For some attributes, after detecting the attribute, it is also necessary to detect the facial area where the attribute is located in order to perform post-processing such as beauty effects. For example, after detecting the eye shape or mouth shape, it is necessary to detect the specific position of the eyes or mouth in order to perform post-processing such as beauty effects on the eyes or mouth.

[0005] 3. Existing face attribute detection methods have high detection accuracy for frontal faces, but cannot guarantee the detection accuracy for faces at other angles;

[0006] Fourth, the existing face attribute detection methods have independent detection processes for each attribute. When the detection accuracy of one attribute is high, this effect does not have any positive impact on the detection accuracy of other attributes. Summary of the invention

[0007] The embodiments of the present invention provide a method and device for establishing a face multi-attribute detection model and a method and device for detecting a face multi-attribute, so as to improve the efficiency of detecting a face multi-attribute.

[0008] The technical solution of the embodiment of the present invention is achieved as follows:

[0009] A method for establishing a multi-attribute face detection model, the method comprising:

[0010] A1. Obtain a training sample set containing multiple frames of face sample images, obtain the true category of each attribute to be detected in each frame of face sample image, and obtain the true position index map of each frame of face sample image;

[0011] A2, reading face sample images from the training sample set in sequence, inputting the read face sample images into the feature encoding module of the face multi-attribute detection model to be trained, and obtaining the encoding features of the face sample images;

[0012] A3, inputting the encoded features of the face sample image into a position index map acquisition module of the face multi-attribute detection model to be trained to obtain a predicted position index map of the face sample image;

[0013] A4, searching for the coding feature of each attribute of the face sample image in the coding feature of the face sample image according to the position of each feature point included in each attribute in the predicted position index map of the face sample image on the face sample image;

[0014] A5. Input the encoding features of each attribute of the face sample image into the classification module of the face multi-attribute detection model to be trained to obtain the predicted category of each attribute;

[0015] A6. Calculate a loss function based on the predicted position index map and the actual position index map of the face sample image, as well as the predicted category and the actual category of each attribute of the face sample image, and use the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module, and the classification module;

[0016] A7. Return to A2 until the face multi-attribute detection model converges, and use the converged face multi-attribute detection model as the final face multi-attribute detection model.

[0017] The step A1 of obtaining the real position index map of each face sample image frame includes:

[0018] For each frame of face sample image, feature points are detected in the face sample image, and the detected feature points are mapped to a standard face position index map to obtain a true position index map of the face sample image; wherein the standard face position index map includes: feature points and triangular pieces composed of feature points, each feature point represents a preset part point of the face, each triangular piece represents a preset area of ​​the face, and each vertex of each triangular piece is a feature point.

[0019] The triangle piece on the standard face position index map is:

[0020] It is obtained by triangulating the standard face based on the feature points detected on the standard face and the preset triangulation principle.

[0021] After mapping the detected feature points to the standard face position index map in step A1 and before obtaining the real position index map of the face sample image, the method further includes:

[0022] Linear interpolation is performed based on multiple triangles in the standard face position index map to expand the number of feature points.

[0023] The step A1 of performing linear interpolation based on multiple triangles in the standard face position index map includes:

[0024] For each preset important attribute, the face area corresponding to the important attribute on the standard face position index map is enlarged, and linear interpolation is performed based on the triangles contained in the enlarged face area.

[0025] The step A1 of obtaining the true category of each attribute to be detected in each frame of the face sample image includes:

[0026] Obtain the true category of each attribute to be detected in each frame of manually calibrated face sample image;

[0027] Alternatively, each frame of the face sample image is input into the trained classification models corresponding to each attribute, so as to obtain the true category of each attribute to be detected in each frame of the face sample image.

[0028] Step A6, calculating the loss function according to the predicted position index map and the real position index map of the face sample image, and the predicted category and the real category of each attribute of the face sample image, includes:

[0029] Calculate a first loss function based on the predicted position index map and the real position index map of the face sample image; calculate a second loss function based on the predicted category and the real category of each attribute of the face sample image;

[0030] The step A6 of using the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module includes:

[0031] The first loss function value is used to update the neural network parameters of the feature encoding module and the position index map acquisition module; the second loss function value is used to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module.

[0032] A method for detecting multi-attributes of a face, the method comprising:

[0033] Inputting the face image to be detected into the feature encoding module of the trained face multi-attribute detection model to obtain the encoding features of the face image;

[0034] Inputting the encoded features of the face image into a position index map acquisition module of a trained face multi-attribute detection model to obtain a position index map of the face image;

[0035] According to the position of each feature point included in each attribute in the position index map of the face image on the face image, searching for the coding feature of each attribute of the face image in the coding features of the face image;

[0036] The encoding features of each attribute of the face image are input into the classification module of the trained face multi-attribute detection model to obtain the category of each attribute.

[0037] A device for establishing a face multi-attribute detection model, the device comprising:

[0038] A training preparation module is used to obtain a training sample set containing multiple frames of face sample images, obtain the true category of each attribute to be detected in each frame of face sample image, and obtain the true position index map of each frame of face sample image;

[0039] A feature encoding module is used to perform feature encoding on the face sample images sequentially read from the training sample set to obtain the encoding features of the face sample images;

[0040] A position index map acquisition module is used to learn a predicted position index map of a face sample image based on the encoding features of the face sample image;

[0041] The attribute coding feature mapping module is used to search the coding features of each attribute of the face sample image in the coding features of the face sample image according to the positions of each feature point contained in each attribute in the predicted position index map of the face sample image on the face sample image;

[0042] The classification module is used to classify each attribute according to the encoding features of each attribute of the face sample image to obtain the predicted category of each attribute;

[0043] The parameter adjustment module is used to calculate the loss function according to the predicted position index map and the actual position index map of the face sample image, as well as the predicted category and the actual category of each attribute of the face sample image, and use the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module until the face multi-attribute detection model converges, and the converged face multi-attribute detection model is used as the final face multi-attribute detection model.

[0044] A multi-attribute face detection device, comprising:

[0045] A feature encoding module is used to perform feature encoding on the input face image to obtain the encoded features of the face image;

[0046] A position index map acquisition module; used for learning the position index map of the face image according to the encoding features of the face image;

[0047] The attribute coding feature mapping module is used to search the coding features of each attribute of the face image in the coding features of the face image according to the positions of the feature points contained in each attribute in the position index map of the face image on the face image;

[0048] The classification module is used to classify each attribute of the face image according to the encoding features of each attribute to obtain the predicted category of each attribute.

[0049] In an embodiment of the present invention, a position index map of a face is predicted by coding features of a whole frame of face sample images, and then coding features corresponding to each attribute are found in the coding features of the whole frame of face sample images by the position index map of the face. The coding features corresponding to each attribute are used to classify each attribute, thereby eliminating the need for additional detection of each area of ​​the face and detecting the categories of all attributes at one time, thereby improving the efficiency of multi-attribute detection of faces; and multi-attribute detection of faces of any shape and angle can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A flowchart of a method for establishing a multi-attribute face detection model provided by an embodiment of the present invention;

[0052] Figure 2 is an example diagram of a standard face position index map when n=68;

[0053] Figure 3 After detecting feature points of a face, the feature points are mapped to a standard face position index map to obtain an example of a real position index map;

[0054] Figure 4 A flowchart of a method for multi-attribute face detection provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of the structure of a device for establishing a multi-attribute face detection model provided by an embodiment of the present invention;

[0056] Figure 6 A schematic diagram of the structure of a face multi-attribute detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0059] The inventor of the present application has found that the segmentation method based on input text can segment the area of ​​interest by inputting words or phrases such as "car" or "blue sky". This method can improve the flexibility and interpretability of the segmentation method, because people only pay attention to specific categories in many scenarios. The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0060] Figure 1 This is a flow chart of a method for establishing a face multi-attribute detection model provided by an embodiment of the present invention, and the specific steps are as follows:

[0061] Step 101: Obtain a training sample set including multiple frames of face sample images, obtain the true category of each attribute to be detected in each frame of face sample image, and obtain the true position index map of each frame of face sample image.

[0062] Attributes such as eye shape, mouth shape, nose shape, hairstyle, whether to wear glasses, or whether to wear a mask, etc. Each attribute corresponds to one or more categories, and the category of each attribute can be set according to conventions or actual needs. For example, the categories of eye shape include eight categories: almond eyes, lychee eyes, phoenix eyes, auspicious phoenix eyes, squinting eyes, triangular eyes, peach blossom eyes, and willow eyes. The categories of mouth shape include six categories: square, arch, heart, thick on top and thin on the bottom, cherry, and M. The categories of nose shape include seven categories: Greek nose, Roman nose, Jewish nose, wavy nose, lion nose, saddle nose, and garlic nose. The categories of hairstyle include three categories: short hair, medium hair, and long hair. The categories of whether to wear glasses include two categories: wearing glasses and not wearing glasses. The categories of whether to wear a mask include two categories: wearing glasses and not wearing a mask.

[0063] When obtaining the true category of each attribute to be detected in each frame of face sample image, the category of each attribute to be detected can be manually identified, and the true category of each attribute to be detected in each frame of face sample image that is manually calibrated is obtained; or, a classification model can be trained in advance for each attribute. At this time, each frame of face sample image is input into the classification models corresponding to the trained attributes respectively, and the true category of each attribute to be detected in each frame of face sample image is obtained.

[0064] In an optional embodiment, obtaining the real position index map of each face sample image frame includes: for each face sample image frame, detecting feature points in the face sample image, mapping the detected feature points to the standard face position index map, and obtaining the real position index map of the face sample image. Among them, the standard face position index map includes: each feature point and a triangular piece composed of each feature point, each feature point represents a preset part point of the face, such as: a feature point corresponds to the left corner of the left eye, a feature point corresponds to the right corner of the left eye, a feature point corresponds to the tip of the nose, etc., each triangular piece represents a preset area of ​​the face, and each vertex of each triangular piece is a feature point. It should be emphasized that the coordinates of each feature point in the real position index map of the face sample image are the coordinates of each feature point on the face sample image. In one embodiment, the triangular piece can be divided according to the principle of Delaunay triangulation algorithm.

[0065] Among them, the triangle pieces on the standard face position index map are obtained by triangulating the standard face according to the feature points detected on the standard face and the preset triangulation principle, each vertex of each triangle piece is a feature point, and each triangle piece represents a preset face area. Since the more feature points there are, the better the final face multi-attribute detection effect is, therefore, in an optional embodiment, after mapping the detected feature points to the standard face position index map and before obtaining the real position index map of the face sample image, it further includes: performing linear interpolation based on multiple triangle pieces in the standard face position index map to expand the number of feature points. The number of extended feature points can be set according to actual needs. Among them, the coordinates of each feature point used in interpolation are the coordinates of each feature point on the face sample image, so that the coordinates of the interpolated and extended feature points are also the coordinates of the feature points on the face sample image.

[0066] In an optional embodiment, linear interpolation is performed based on multiple triangles in the standard face position index map, including: for each pre-set important attribute, the corresponding face area of ​​the important attribute on the standard face position index map is enlarged, and linear interpolation is performed based on the triangles contained in the enlarged face area. In this way, for the important attribute, after the corresponding face area on the standard face position index map is enlarged, the number of extended feature points is greater, thereby ultimately making the classification result of the attribute more accurate.

[0067] The process of obtaining the standard face position index map is described below:

[0068] The front of a preset standard 3D (3-dimensional) face model is projected onto a 2D plane. Suppose the number of points projected onto the 2D plane is n points, and the n points are feature points, where each feature point uniquely represents a facial part point, for example: feature point m (1≤m≤n) represents the left corner of the left eye of the face; then, the preset triangulation principle (i.e. which three feature points constitute a triangle) is used to triangulate the n points, and at this time each triangle can represent an area of ​​the face, and a standard face position index map is obtained.

[0069] Figure 2 : is an example of a standard face position index map when n=68, where 1 to 17 of the 68 feature points are face contour points, 18 to 22, 23 to 27 are right and left eyebrow contour points, 28 to 31 are nose midline points, 32 to 36 are nose bottom contour points, 37 to 42, 43 to 48 are right and left eye contour points, 49 to 60 are mouth outer contour points, and 61 to 68 are mouth inner contour points. The specific part represented by each feature point is as follows: Figure 2 As shown, such as: 1 represents the upper endpoint of the right ear on the face, 17 represents the upper endpoint of the left ear on the face, 9 represents the tip of the chin, 18 represents the right endpoint of the right eyebrow (the left and right of the picture and the actual face are opposite), 22 represents the left endpoint of the right eyebrow, 28 represents the vertex of the midline of the nose, 31 represents the tip of the nose, 37 represents the right corner of the right eye, 40 represents the left corner of the right eye, 49 represents the right outer corner of the mouth, 55 represents the left outer corner of the mouth, 61 represents the right inner corner of the mouth, 65 represents the left inner corner of the mouth, and so on; the triangle piece after triangulation based on 68 feature points is as follows Figure 2 shown.

[0070] The real position index map of the face sample image can be described by the following information: the 2D coordinates of each feature point on the face sample image, the association relationship between each triangle and the feature points it contains, and the association relationship between each attribute and the triangle it contains. The feature points contained in each triangle include: three feature points corresponding to the three vertices of the triangle and each feature point contained in the triangle.

[0071] Among them, a unique feature point serial number can be assigned to each feature point, a unique triangle serial number can be assigned to each triangle, and a unique attribute serial number can be assigned to each attribute. Then: the association relationship between each triangle and the feature points it contains can be: the triangle serial number of each triangle is associated with the feature point serial number of each feature point contained in the triangle, and the association relationship between each attribute and the triangles it contains can be: the attribute serial number of each attribute is associated with the triangle serial number of each triangle contained in the attribute. It should be pointed out that when assigning a serial number to each feature point, each triangle and each attribute, its serial number represents the facial part point corresponding to the feature point, or the facial area corresponding to the triangle, or the attribute corresponding to the attribute.

[0072] Figure 3 After detecting feature points of a face, the feature points are mapped to the standard face position index map to obtain an example of a real position index map. The left picture is an example of a face with feature points detected, and the right picture is an example of a real position index map obtained by mapping the feature points in the left picture to the standard face position index map. Since the mouth of the face is closed, the feature points 62-64 and 66-68 of the upper and lower lips overlap, so there are only 65 feature points in total.

[0073] Step 102: Read sample face images from the training sample set in sequence, input the read sample face images into the feature encoding module of the face multi-attribute detection model to be trained, and obtain the encoding features of the sample face images.

[0074] The feature encoding module can use one or any combination of convolutional neural network, residual neural network, etc. The number of network layers and the size of each network layer can be determined according to the capabilities of the chip running the face multi-attribute detection model and the required face multi-attribute detection accuracy.

[0075] Step 103: Input the encoded features of the face sample image into the position index map acquisition module of the face multi-attribute detection model to be trained to obtain the predicted position index map of the face sample image.

[0076] The description information of the predicted position index map is consistent with the description information of the real position index map, including: the predicted 2D coordinates of each feature point on the face sample image on the face sample image, the predicted association relationship between each triangle of the face sample image and the feature points it contains, and the predicted association relationship between each attribute and the triangle it contains. The feature points contained in each triangle include: three feature points corresponding to the three vertices of the triangle and each feature point contained in the triangle.

[0077] Among them, each feature point can be assigned a unique feature point serial number, each triangle can be assigned a unique triangle serial number, and each attribute can be assigned a unique attribute serial number. Then: the association relationship between each triangle and the feature points it contains can be: the triangle serial number of each triangle is associated with the feature point serial numbers of each feature point contained in the triangle, and the association relationship between each attribute and the triangles it contains can be: the attribute serial number of each attribute is associated with the triangle serial number of each triangle contained in the attribute. It should be noted that when assigning feature point serial numbers to feature points in the predicted position index map, the assignment principle is the same as the assignment principle for assigning feature point serial numbers to feature points in the real position index map, that is, the serial numbers assigned to feature points corresponding to the same facial part in the predicted position index map and the real position index map are the same, for example: the feature point corresponding to the left outer corner of the mouth in the predicted position index map and the feature point corresponding to the left outer corner of the mouth in the real position index map have the same serial numbers; similarly, when assigning triangle serial numbers to triangles in the predicted position index map, the assignment principle is the same as the assignment principle for triangle serial numbers to triangles in the real position index map, that is, the serial numbers assigned to triangles corresponding to the same facial area in the predicted position index map and the real position index map are the same; similarly, when assigning attribute serial numbers to attributes in the predicted position index map, the assignment principle is the same as the assignment principle for assigning attribute serial numbers to attributes in the real position index map, that is, the serial numbers assigned to attributes corresponding to the same attribute in the predicted position index map and the real position index map are the same.

[0078] Step 104: searching for the encoding features of each attribute of the face sample image in the encoding features of the face sample image according to the positions of the feature points contained in each attribute in the predicted position index map of the face sample image on the face sample image.

[0079] Among them, according to the description information of the predicted position index map: the predicted 2D coordinates of each feature point on the face sample image on the face sample image, the predicted association between each triangle piece of the face sample image and the feature points it contains, and the predicted association between each attribute and the triangle piece it contains, the 2D coordinates of each feature point contained in each attribute in the predicted position index map of the face sample image on the face sample image can be known.

[0080] The coding feature of the face sample image is essentially a matrix, and each element in the matrix corresponds to an area of ​​the face sample image, that is, each element is used to represent the features of an area of ​​the face sample image. Therefore, after knowing the positions of the feature points contained in each attribute of the face sample image, it is possible to know which coding features each attribute corresponds to.

[0081] Step 105: Input the encoding features of each attribute of the face sample image into the classification module of the face multi-attribute detection model to be trained to obtain the predicted category of each attribute.

[0082] The classification module usually adopts a fully connected network. The specific structure of the fully connected network can be determined based on multiple experiments and / or actual needs (such as the capabilities of the chip running the face multi-attribute detection model and the required face multi-attribute detection accuracy, etc.).

[0083] Step 106: Calculate the loss function based on the predicted position index map and the actual position index map of the face sample image, as well as the predicted category and the actual category of each attribute of the face sample image, and use the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module.

[0084] In an optional embodiment, a loss function is calculated based on the predicted position index map and the actual position index map of the face sample image, and the predicted category and the actual category of each attribute of the face sample image, including: calculating a first loss function based on the predicted position index map and the actual position index map of the face sample image; calculating a second loss function based on the predicted category and the actual category of each attribute of the face sample image.

[0085] At the same time, the loss function value is used to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module, including: using the first loss function value to update the neural network parameters of the feature encoding module and the position index map acquisition module; using the second loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module.

[0086] Step 107: Return to step 102 until the face multi-attribute detection model converges, and use the converged face multi-attribute detection model as the final face multi-attribute detection model.

[0087] In the above embodiment, the position index map of the face is first predicted by the coding features of the whole frame face sample image, and then the coding features corresponding to each attribute are found in the coding features of the whole frame face sample image by the position index map of the face, and each attribute is classified by using the coding features corresponding to each attribute, so that there is no need to perform additional detection of each area of ​​the face, and the categories of all attributes can be detected at one time, thereby improving the efficiency of multi-attribute detection of the face; and, when the position index map is predicted well, the extraction effect of the coding features of each attribute is better, and thus the classification result of each attribute is better, which in turn stimulates the accuracy of the extraction of the coding features of each attribute, and further improves the accuracy of the position index map prediction, that is, different attributes can complement each other to promote learning; and, multi-attribute detection of faces of any shape and any angle can be achieved.

[0088] Figure 4 This is a flowchart of a face multi-attribute detection method provided by an embodiment of the present invention, and the specific steps are as follows:

[0089] Step 401: input the face image to be detected into the feature encoding module of the trained face multi-attribute detection model to obtain the encoding features of the face image.

[0090] Step 402: Input the encoded features of the face image into a position index map acquisition module of a trained face multi-attribute detection model to obtain a position index map of the face image.

[0091] Step 403: searching for the encoding feature of each attribute of the face image in the encoding features of the face image according to the positions of the feature points contained in each attribute in the position index map of the face image on the face image.

[0092] Step 404: Input the encoding features of each attribute of the face image into the classification module of the trained face multi-attribute detection model to obtain the category of each attribute.

[0093] In the above embodiment, the position index map of the face is first obtained through the coding features of the whole frame face image, and then the coding features corresponding to each attribute are found in the coding features of the whole frame face image through the position index map of the face, and the coding features corresponding to each attribute are used to classify each attribute, thereby eliminating the need for additional detection of each area of ​​the face, and the categories of all attributes can be detected at one time, thereby improving the efficiency of multi-attribute detection of the face; and multi-attribute detection of faces of any shape and any angle can be achieved.

[0094] The execution subject of the embodiment of the present invention may be a GPU (Graphics Processing Unit).

[0095] Figure 5 The schematic diagram of the structure of the apparatus for establishing a multi-attribute face detection model provided by an embodiment of the present invention mainly comprises: a training preparation module 51, a feature encoding module 52, a position index map acquisition module 53, an attribute encoding feature mapping module 54, a classification module 55 and a parameter adjustment module 56, wherein:

[0096] The training preparation module 51 is used to obtain a training sample set including multiple frames of face sample images, obtain the true category of each attribute to be detected in each frame of face sample image, obtain the true position index map of each frame of face sample image, and send the true category of each attribute to be detected in each frame of face sample image and the true position index map of each frame of face sample image to the parameter adjustment module 56.

[0097] The feature encoding module 52 is used to read face sample images from the training sample set in sequence, perform feature encoding on the read face sample images, obtain the encoded features of the face sample images, and send the encoded features of the face sample images to the position index map acquisition module 53 and the attribute encoding feature mapping module 54.

[0098] The position index map acquisition module 53 is used to learn the predicted position index map of the face sample image according to the encoding features of the face sample image, and send the predicted position index map of the face sample image to the attribute encoding feature mapping module 54 and the parameter adjustment module 56.

[0099] The attribute coding feature mapping module 54 is used to search for the coding features of each attribute of the face sample image in the coding features of the face sample image according to the positions of the feature points contained in each attribute in the predicted position index map of the face sample image on the face sample image, and send the coding features of each attribute of the face sample image to the classification module 55.

[0100] The classification module 55 is used to classify each attribute of the face sample image according to the encoding features of each attribute, obtain the predicted category of each attribute, and send the predicted category of each attribute to the parameter adjustment module 56.

[0101] The parameter adjustment module 56 is used to calculate the loss function according to the predicted position index map and the actual position index map of the face sample image, as well as the predicted category and the actual category of each attribute of the face sample image, and use the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module until the face multi-attribute detection model converges, and use the converged face multi-attribute detection model as the final face multi-attribute detection model.

[0102] In an optional embodiment, the training preparation module 51 obtains the real position index map of each face sample image, including: for each face sample image, detecting feature points in the face sample image, mapping the detected feature points to the standard face position index map, and obtaining the real position index map of the face sample image. The standard face position index map includes: each feature point and a triangular piece composed of each feature point, each feature point represents a preset part point of the face, each triangular piece represents a preset area of ​​the face, and each vertex of each triangular piece is a feature point.

[0103] In an optional embodiment, the triangular pieces on the standard face position index map used by the training preparation module 51 are obtained by triangulating the standard face according to the feature points detected on the standard face and the preset triangulation principle. In an optional embodiment, after the training preparation module 51 maps the detected feature points to the standard face position index map and before obtaining the real position index map of the face sample image, it further includes: performing linear interpolation based on multiple triangular pieces in the standard face position index map to expand the number of feature points.

[0104] In an optional embodiment, the training preparation module 51 performs linear interpolation based on multiple triangles in the standard face position index map, including: for each pre-set important attribute, enlarging the corresponding face area of ​​the important attribute on the standard face position index map, and performing linear interpolation based on the triangles contained in the enlarged face area.

[0105] In an optional embodiment, the training preparation module 51 obtains the true category of each attribute to be detected in each frame of the face sample image, including: obtaining the true category of each attribute to be detected in each frame of the face sample image that is manually calibrated; or, inputting each frame of the face sample image into the classification models corresponding to each trained attribute, respectively, to obtain the true category of each attribute to be detected in each frame of the face sample image.

[0106] In an optional embodiment, the parameter adjustment module 56 calculates the loss function according to the predicted position index map and the real position index map of the face sample image, and the predicted category and the real category of each attribute of the face sample image, including: calculating the first loss function according to the predicted position index map and the real position index map of the face sample image; calculating the second loss function according to the predicted category and the real category of each attribute of the face sample image;

[0107] Moreover, the parameter adjustment module 56 uses the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module, including: using the first loss function value to update the neural network parameters of the feature encoding module and the position index map acquisition module; using the second loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module.

[0108] Figure 6 The schematic diagram of the structure of the multi-attribute face detection device provided by the embodiment of the present invention mainly comprises: a feature encoding module 61, a position index map acquisition module 62, an attribute encoding feature mapping module 63 and a classification module 64, wherein:

[0109] The feature encoding module 61 is used to perform feature encoding on the input face image to obtain the encoded features of the face image, and send the encoded features of the face image to the position index map acquisition module 62 and the attribute encoding feature mapping module 63.

[0110] The position index map acquisition module 62 is used to learn the position index map of the face image according to the encoding features of the face image, and send the position index map of the face image to the attribute encoding feature mapping module 63.

[0111] The attribute coding feature mapping module 63 is used to search the coding features of each attribute of the face image in the coding features of the face image according to the positions of the feature points contained in each attribute in the position index map of the face image on the face image.

[0112] The classification module 64 is used to classify each attribute of the face image according to the encoding features of each attribute to obtain the predicted category of each attribute.

[0113] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores instructions. When the instructions are executed by a processor, the processor causes the processor to execute the steps of the method for establishing a multi-attribute face detection model described in any embodiment of the present invention, or execute the steps of the multi-attribute face detection method described in any embodiment of the present invention.

[0114] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0115] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.

Claims

1. A method for establishing a multi-attribute face detection model, characterized in that: The method includes: A1. Obtain a training sample set containing multiple frames of face sample images, obtain the true category of each attribute to be detected in each frame of face sample image, and obtain the true position index map of each frame of face sample image; A2, reading face sample images from the training sample set in sequence, inputting the read face sample images into the feature encoding module of the face multi-attribute detection model to be trained, and obtaining the encoding features of the face sample images; A3, inputting the encoded features of the face sample image into a position index map acquisition module of the face multi-attribute detection model to be trained to obtain a predicted position index map of the face sample image; A4, searching for the coding feature of each attribute of the face sample image in the coding feature of the face sample image according to the position of each feature point included in each attribute in the predicted position index map of the face sample image on the face sample image; A5. Input the encoding features of each attribute of the face sample image into the classification module of the face multi-attribute detection model to be trained to obtain the predicted category of each attribute; A6. Calculate a loss function based on the predicted position index map and the actual position index map of the face sample image, as well as the predicted category and the actual category of each attribute of the face sample image, and use the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module, and the classification module; A7. Return to A2 until the face multi-attribute detection model converges, and use the converged face multi-attribute detection model as the final face multi-attribute detection model.

2. The method according to claim 1, characterized in that The step A1 of obtaining the real position index map of each face sample image frame includes: For each frame of face sample image, feature points are detected in the face sample image, and the detected feature points are mapped to a standard face position index map to obtain a real position index map of the face sample image; wherein the standard face position index map includes: feature points and triangular pieces composed of feature points, each feature point represents a preset part point of the face, each triangular piece represents a preset area of ​​the face, and each vertex of each triangular piece is a feature point.

3. The method according to claim 2, characterized in that The triangle piece on the standard face position index map is: It is obtained by triangulating the standard face based on the feature points detected on the standard face and the preset triangulation principle.

4. The method according to claim 2, characterized in that: After mapping the detected feature points to the standard face position index map in step A1 and before obtaining the real position index map of the face sample image, the method further includes: Linear interpolation is performed based on multiple triangles in the standard face position index map to expand the number of feature points.

5. The method according to claim 4, characterized in that The linear interpolation based on multiple triangles in the standard face position index map in step A1 includes: For each preset important attribute, the face area corresponding to the important attribute on the standard face position index map is enlarged, and linear interpolation is performed based on the triangles contained in the enlarged face area.

6. The method according to claim 1, characterized in that The step A1 of obtaining the true category of each attribute to be detected in each frame of the face sample image includes: Obtain the true category of each attribute to be detected in each frame of manually calibrated face sample image; Alternatively, each frame of the face sample image is input into the trained classification models corresponding to each attribute, so as to obtain the true category of each attribute to be detected in each frame of the face sample image.

7. The method according to claim 1, characterized in that Step A6, calculating the loss function according to the predicted position index map and the real position index map of the face sample image, and the predicted category and the real category of each attribute of the face sample image, includes: Calculate a first loss function based on the predicted position index map and the real position index map of the face sample image; calculate a second loss function based on the predicted category and the real category of each attribute of the face sample image; The step A6 of using the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module includes: The first loss function value is used to update the neural network parameters of the feature encoding module and the position index map acquisition module; the second loss function value is used to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module.

8. A multi-attribute face detection method, characterized in that: The method includes: Inputting the face image to be detected into the feature encoding module of the trained face multi-attribute detection model to obtain the encoding features of the face image; Inputting the encoded features of the face image into a position index map acquisition module of a trained face multi-attribute detection model to obtain a position index map of the face image; According to the position of each feature point included in each attribute in the position index map of the face image on the face image, searching for the coding feature of each attribute of the face image in the coding features of the face image; The encoding features of each attribute of the face image are input into the classification module of the trained face multi-attribute detection model to obtain the category of each attribute.

9. A device for establishing a multi-attribute face detection model, characterized in that: The device includes: A training preparation module is used to obtain a training sample set containing multiple frames of face sample images, obtain the true category of each attribute to be detected in each frame of face sample image, and obtain the true position index map of each frame of face sample image; A feature encoding module is used to perform feature encoding on the face sample images sequentially read from the training sample set to obtain the encoding features of the face sample images; A position index map acquisition module is used to learn a predicted position index map of a face sample image based on the encoding features of the face sample image; The attribute coding feature mapping module is used to search the coding features of each attribute of the face sample image in the coding features of the face sample image according to the positions of each feature point contained in each attribute in the predicted position index map of the face sample image on the face sample image; The classification module is used to classify each attribute according to the encoding features of each attribute of the face sample image to obtain the predicted category of each attribute; The parameter adjustment module is used to calculate the loss function according to the predicted position index map and the actual position index map of the face sample image, as well as the predicted category and the actual category of each attribute of the face sample image, and use the loss function value to update the neural network parameters of the feature encoding module, the position index map acquisition module and the classification module until the face multi-attribute detection model converges, and the converged face multi-attribute detection model is used as the final face multi-attribute detection model.

10. A multi-attribute face detection device, characterized in that: The device includes: A feature encoding module is used to perform feature encoding on the input face image to obtain the encoded features of the face image; A position index map acquisition module; used for learning the position index map of the face image according to the encoding features of the face image; The attribute coding feature mapping module is used to search the coding features of each attribute of the face image in the coding features of the face image according to the positions of the feature points contained in each attribute in the position index map of the face image on the face image; The classification module is used to classify each attribute of the face image according to the encoding features of each attribute to obtain the predicted category of each attribute.

Images