An age recognition method based on multi-feature fusion of 3D face image

Abstract

The application relates to an age recognition method based on 3D face image multi-feature fusion, which comprises the following steps: filtering background noise in a 2D image according to depth information in a depth image; extracting age, gender and race features in the 2D image from which the background noise is filtered by using multiple convolutional neural networks; automatically learning weight parameters of influences of the age, gender and race features on age recognition based on an attention concentration mechanism, and then fusing the features according to respective parameters for age prediction. A background noise filtering method is provided, depth information of a face is calculated by using depth image information, background noise in a 2D image is filtered according to a face depth range, so that only a face image area is reserved, and the accuracy of age recognition is improved; multiple convolutional neural networks are used to extract age, gender and race features, and an adaptive feature fusion strategy is provided, the features are fused into age features with higher robustness, and parameters are adaptively adjusted.

Description

TECHNICAL FIELD

[0001] The present application relates to the field, more particularly, to an age recognition method based on multi-feature fusion of 3D face images. BACKGROUND

[0002] Age recognition is to predict the age information of a face in an image, which has been a hot topic in the field of artificial intelligence for a long time, and has very wide and practical applications, such as in human-computer interaction, intelligent robots can make more personalized responses according to users of different ages; for example, when searching for suspicious personnel through monitoring videos, the search range can be quickly narrowed down according to age. Most of the traditional age recognition methods are based on 2D face images, and 2D images are easily affected by changes in light and head posture, resulting in a large deviation in the predicted age of the same person under different light or different head posture. In addition, most of the traditional age recognition methods use convolutional neural networks to extract single age features for age recognition, ignoring the influence of different genders and races on age features, but in fact different genders and races have different age features. Therefore, it is still necessary to explore an age recognition method that can overcome changes in light, posture, gender and race.

[0003] In the prior art, a Chinese invention patent discloses a face age recognition method, device, system, electronic device and storage medium, wherein the face age recognition method acquires a face frame by performing face detection on a collected image; detects feature points in the face frame, and judges a face expression according to the detected feature points; in the case that the face expression meets the recognition condition, judges the age classification of the face in the collected image through an age recognition model, solves the problem of low accuracy of age recognition through a face, and improves the accuracy and reliability of age recognition through a face only, but ignores the influence of different genders and races on age features. SUMMARY

[0004] The present application provides an age recognition method based on multi-feature fusion of 3D face images to solve the technical defects of the prior art age recognition methods that ignore the influence of different genders and races on age features.

[0005] To achieve the above application purposes, the technical solutions adopted are as follows:

[0006] An age recognition method based on multi-feature fusion of 3D face images, comprising the following steps:

[0007] S1: filtering out background noise in a 2D image according to depth information in a depth image;

[0008] S2: using multiple convolutional neural networks to respectively extract age, gender and race features in the 2D image from which the background noise has been filtered out;

[0009] S3: Automatically learning the weight parameters of the influence of age, gender and race features on age recognition based on an attention focusing mechanism, and fusing the features according to the respective parameters for age prediction.

[0010] In the above scheme, a new background noise filtering method is proposed, the range of face depth information is calculated using the depth image information in the RGB-D image, and the background noise in the 2D image is filtered out according to the face depth range, so that only the face image area is retained, the interference of noise is reduced, and the accuracy of age recognition is improved; multiple convolutional neural networks are used to extract age, gender and race features, and an adaptive feature fusion strategy is proposed to fuse these features into more robust age features, so that the model can adaptively adjust the parameters according to the differences in gender and race when predicting age, thereby improving the flexibility of the model.

[0011] Preferably, in step S1, the following sub-steps are included:

[0012] S11: Traverse the entire 2D image using local binary pattern features to detect the preliminary location of the face;

[0013] S12: Crop the corresponding face area in the step S1 area, and calculate the histogram of pixel values on the face area;

[0014] S13: According to the face depth information of the histogram, get the face image filtered out of the background noise.

[0015] In the above scheme, when shooting outdoors, the face image often contains complex background, and the convolutional neural network will consider the background as part of the face when learning face features, so these background noises may interfere with the feature extraction of the convolutional neural network, resulting in a decrease in age recognition accuracy. In the 2D (RGB) image, there is no obvious difference in pixel value between the face and the background, but in the depth map (D), the face and the background are located in different positions, so the depth information of the face and the background noise is obviously different, so the background noise in the corresponding 2D image can be filtered out according to the depth information of the face in the depth map.

[0016] Preferably, in step S11, the local binary pattern feature is to traverse the entire image row by row and column by column using an LBP operator (3x3), thereby calculating the feature map of the image.

[0017] Preferably, in step S12, after obtaining the face area, there is still background noise in the area, and the range of pixel value concentration is compared according to the size of the face area, to obtain the histogram of pixel values.

[0018] Preferably, in step S13, the 3D face image is defined as [I rgb (i,j),I d (i,j)],where I grb (x,y) is a 2D face image, I d (i,j) is the corresponding depth information image, the face depth information calculated according to step S12, and the background noise is filtered out, where μ is the maximum component in the histogram, v is the range length, Pixel is the original pixel value, and Pixel' is the filtered pixel value.

[0019] Preferably, the depth image is obtained by using a Kinect camera, and the Kinect camera is used to obtain the depth information; and the background noise is filtered out by the following formula:

[0020]

[0021] Each pixel point of the face depth image is traversed, if the pixel value of the point is outside the range of [μ-v, μ+v], the pixel value of the corresponding point in the 2D image is set to 0; if the pixel of the point is within the range of [μ-v, μ+v], the pixel value of the corresponding point in the 2D image remains unchanged; after traversing each pixel point, the face image after filtering out the background noise can be obtained; where μ is the maximum component in the histogram, v is the range length, Pixel is the original pixel value, and Pixel' is the filtered pixel value.

[0022] Preferably, in step S2, after the features are extracted by the convolution layer of the convolutional neural network, a nonlinear ReLU activation function is used to activate the features extracted by the convolution, which can suppress the negative output of the convolution layer and amplify the positive output of the convolution layer; after the activation function, a batch normalization layer Batch Normal is used to normalize the feature output of multiple samples; after the normalization layer, the features are input to the pooling layer to extract the most representative features, while reducing the number of features and improving the calculation speed.

[0023] In the above scheme, the three convolutional neural networks have the same structure, each convolutional neural network includes four convolutional blocks, each convolutional block includes one convolution layer, one nonlinear activation function, one batch normalization layer and one pooling layer. The number of convolution kernels in each of the four convolutional blocks is 16, 32, 64 and 128 respectively, the size of the convolution kernel is 3x3, and the step is set to 1.

[0024] Preferably, the age, race and gender features are obtained by training three networks respectively with three different loss functions; the pictures are labeled with the above three information and then put into the network for training, and finally the above features are learned; in the gender convolutional network, the output is a two-bit one-dimensional vector, [0, 1] represents male, and [1, 0] represents female; in the race convolutional network, the output is a three-bit one-dimensional vector, [0, 0, 1] represents Asian, [0, 1, 0] represents European and American, and [1, 0, 0] represents African; through continuous training of the convolutional neural network, the gender and race features of the face are learned and saved in the convolutional neural network, and the gender and race features are extracted.

[0025] Preferably, in step S3, (F a ),(F g ),(F r )∈R c×w×h are the age, gender and race features respectively, where c is the number of channels of the feature map, and w x h is the size of the feature map; (F a ),(F g ),(F r ) are converted into (F' a ),(F' g ),(F' r )∈R c×1×1 after passing through the CMP layer, and then input to the fully connected layer (FC) and the activation function layer (ReLU), and the outputs (A a ),(A g ),(A r )∈R c×1×1 are one-dimensional column vectors, respectively representing the age, gender and race feature internal attention concentration distribution map, i.e., the weight value of each channel feature map; after connecting (A a ),(A g ),(A r ) by channels, they are input to the fully connected layer (FC) and the activation function layer (ReLU) again to obtain the overall attention concentration distribution map, which is then convolved with the original combined feature map [F a ,F g ,F r ]∈R 3c×w×h to obtain the adaptive fusion feature map (F f ); the adaptive fusion feature map F f is used for age recognition, which automatically adjusts the influence factors of the age, gender and race features on the age prediction result as a whole, so that the model adjusts the weights according to the gender and race information of the face when predicting the age.

[0026] In the above scheme, the degree of influence of age, gender and race characteristics on age recognition is different, and if these characteristics are evenly integrated, it does not conform to the actual situation, therefore, the application proposes an adaptive feature fusion strategy, which automatically learns the degree of influence of each feature on age recognition, and assigns different weights to each feature, so that the model can make flexible adjustments according to different genders and races when predicting age.

[0027] Preferably, the CMP is a channel maximum feature pooling layer, which is used to select the maximum feature value of each dimension channel in the feature map.

[0028] Compared with the prior art, the application has the following beneficial effects:

[0029] The application provides an age recognition method based on 3D face image multi-feature fusion, a new background noise filtering method is proposed, the range of face depth information is calculated by using the depth image information in the RGB-D image, the background noise in the 2D image is filtered according to the face depth range, so that only the face image area is retained, the interference of noise is reduced, and the accuracy of age recognition is improved; a plurality of convolutional neural networks are used to extract age, gender and race features respectively, and an adaptive feature fusion strategy is proposed, the features are fused into more robust age features, so that the model can adaptively adjust the parameters according to the different genders and races when predicting the age, thereby improving the flexibility of the model. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The method flowchart of the application;

[0031] Figure 2 The feature extraction diagram of the LBP operator of the application;

[0032] Figure 3 The depth face of the application;

[0033] Figure 4 The histogram of the depth face image of the application;

[0034] Figure 5 The basic flowchart of the adaptive feature fusion strategy of the application;

[0035] Figure 6 The basic flowchart of the application. DETAILED DESCRIPTION

[0036] The drawings are only used for illustrative description, and cannot be understood as a limitation on the patent;

[0037] The application will be further described below in combination with the drawings and examples.

[0038] Example 1

[0039] As Figures 1-5 shown, an age recognition method based on 3D face image multi-feature fusion, comprising the following steps:

[0040] S1: filtering out the background noise in the 2D image according to the depth information in the depth image;

[0041] S2: using multiple convolutional neural networks to extract age, gender and race features in the 2D image after filtering out the background noise;

[0042] S3: based on the attention focusing mechanism, automatically learning the weight parameters of the influence of age, gender and race features on age recognition, and then fusing the above features according to the respective parameters for age prediction.

[0043] In the above scheme, a new background noise filtering method is proposed, which calculates the range of face depth information using the depth image information in the RGB-D image, and filters out the background noise in the 2D image according to the face depth range, so as to only keep the face image area, reduce the interference of noise, and improve the accuracy of age recognition; multiple convolutional neural networks are used to extract age, gender and race features, and an adaptive feature fusion strategy is proposed to fuse these features into more robust age features, so that the model can adaptively adjust the parameters according to the differences in gender and race when predicting age, thereby improving the flexibility of the model.

[0044] Preferably, in step S1, the following sub-steps are included:

[0045] S11: using local binary pattern feature to traverse the entire 2D image to detect the preliminary location of the face;

[0046] S12: cropping the corresponding face area in the preliminary location, and calculating the histogram of pixel values on the face area;

[0047] S13: obtaining the face image after filtering out the background noise according to the face depth information of the histogram.

[0048] In the above scheme, when shooting outdoors, the face image often contains complex background. When learning face features, the convolutional neural network will regard the background as part of the face, so these background noises may interfere with the feature extraction of the convolutional neural network, resulting in a decrease in the accuracy of age recognition. In the 2D(RGB) image, there is no obvious pixel value difference between the face and the background, but in the depth map(D), the face and the background are located in different positions, so the depth information of the face and the background noise is obviously different, so the background noise in the corresponding 2D image can be filtered out according to the depth information of the face in the depth map.

[0049] Preferably, in step S11, the local binary pattern feature is to traverse the whole image row by row and column by column with an LBP operator (3x3), so as to calculate the feature map of the image.

[0050] In the above scheme, as shown in the formula (1), the center of the LBP operator is taken as a threshold value, and the values of the surrounding eight pixels are compared with the threshold value respectively, and 1 is taken if the value is greater than the threshold value, and 0 is taken if the value is less than the threshold value, so as to calculate the feature value of the center point in the 2D image. After traversing all the points of the image, the feature map of the image can be calculated, and then the feature map is input into the face detector to obtain the area range of the face. Figure 2

[0051] Preferably, in step S12, after the face area is obtained, there is still background noise in the area, and the range of the concentrated pixel values is compared according to the size of the area occupied by the face, so as to obtain the histogram of the pixel values.

[0052] In the above scheme, as shown in the formula (2), it can be observed that the face occupies most of the area, and the background occupies a small part. Figure 3 As shown in the formula (3), it can be observed that most of the pixel values are concentrated in a range [200, 230], and a small part of the pixel values are concentrated in another range [0, 30]. Therefore, the main component in the histogram is the depth information of the face, that is, the range in the red dashed line is the main depth information range of the face, and the range outside the range can be considered as background noise. Figure 4

[0053] Preferably, in step S13, the 3D face image is defined as [I rgb (x,y),I d (i,j)], wherein I grb (x,y) is a 2D face image, I d (i,j) is a corresponding depth information image, which is calculated according to the face depth information obtained in step S12, and the background noise is filtered out, wherein μ is the maximum component in the histogram, v is the range length, Pixel is the original pixel value, and Pixel' is the pixel value after filtering.

[0054] Preferably, the depth image is obtained by using a Kinect camera, and the Kinect camera is used to obtain the depth information. The background noise is filtered out by the following formula:

[0055]

[0056] ​​Traverse each pixel point of the face depth image, if the pixel value of the point is out of the range of [μ-ν, μ+ν], set the pixel value of the corresponding point in the 2D image to 0; if the pixel of the point is within the range of [μ-ν, μ+ν], the pixel value of the corresponding point in the 2D image remains unchanged; after traversing each pixel point, the face image after filtering out the background noise can be obtained; wherein μ is the maximum component in the histogram, ν is the range length, Pixel is the original pixel value, and Pixel' is the pixel value after filtering.

[0057] Preferably, in step S2, after the convolutional layer of the convolutional neural network extracts the features, a nonlinear ReLU activation function is used to activate the convolutional extracted features, which can suppress the negative output of the convolutional layer and amplify the positive output of the convolutional layer; after the activation function, the batch normalization layer Batch Normal is used to normalize the feature output of multiple samples; after the normalization layer, the features are input to the pooling layer to extract the most representative features and reduce the number of features to improve the calculation speed.

[0058] In the above scheme, the three convolutional neural networks all adopt the same structure, each convolutional neural network includes four convolutional blocks, and each convolutional block includes one convolutional layer, one nonlinear activation function, one batch normalization layer and one pooling layer. The number of convolutional kernels in each of the four convolutional blocks is 16, 32, 64 and 128 respectively, the size of the convolutional kernel is 3x3, and the step is set to 1.

[0059] Preferably, the age, race and gender features are obtained by training three networks (gender, age, race) with three different loss functions respectively. The picture with the label of the three information, such as (male, 24 years old, Asian), is put into the network for training, and the network can learn these features slowly; in the gender convolutional network, the output is a two-bit one-dimensional vector, [0, 1] represents male, and [1, 0] represents female; in the race convolutional network, the output is a three-bit one-dimensional vector, [0, 0, 1] represents Asian, [0, 1, 0] represents European and American, and [1, 0, 0] represents African; through continuous training of the convolutional neural network, the gender and race features of the face are learned and saved in the convolutional neural network, and the gender and race features are extracted.

[0060] Preferably, in step S3, as shown in Figure 5 , (F a ),(F g ),(F r )∈R c×w×h are the age, gender and race features respectively, wherein c is the number of channels of the feature map, and w x h is the size of the feature map; (F a ),(F g ),(F r) respectively after the CMP layer, and are converted into (F a ),(F g ),(F r )∈R c×1×1 , and are input into a full connection layer (FC) and an activation function layer (ReLU), and outputs are (A a ),(A g ),(A r )∈R c×1×1 are one-dimensional column vectors, respectively representing the age, gender and race feature internal attention concentration distribution map, i.e., the weight value of each channel feature map; (A a ),(A g ),(A r ) are connected by channels and then input into a full connection layer (FC) and an activation function layer (ReLU) to obtain the overall attention concentration distribution map, which is combined with the original feature map [F a ,F g ,F r ]∈R 3c×w×h to obtain the adaptive fusion feature map (F f ), and the adaptive fusion feature map F f is used for age recognition, which automatically adjusts the influence factors of the age, gender and race features on the age prediction result as a whole, so that the model can make weight adjustment according to the gender and race information of the face when predicting the age.

[0061] In the above scheme, since the degrees of influence of the age, gender and race features on age recognition are different, if these features are evenly fused, it does not conform to the actual situation, therefore, the application proposes an adaptive feature fusion strategy, which automatically learns the degree of influence of each feature on age recognition, and assigns different weights to each feature, so that the model can make flexible adjustment according to different genders and races when predicting the age.

[0062] Preferably, the CMP is a channel maximum feature pooling layer, which is used to select the maximum feature value of each dimension channel in the feature map.

[0063] Embodiment 2

[0064] The basic flow of the application is as follows Figure 6As shown, it can be divided into three steps, which are background filtering, multi-feature extraction and adaptive feature fusion, wherein the 3D face image storage format is RGB-D format, RGB is a 2D image, D is a corresponding depth image, the depth image is obtained through a Kinect sensor, the principle thereof is that the Kinect sensor emits infrared rays, the infrared rays are reflected when encountering an obstacle, the sensor records the time of flight (TOF) of the infrared rays, and finally a depth image is generated according to the TOF. For background filtering, the purpose is to eliminate the background noise in the 2D image and improve the accuracy of feature extraction. Since there is no obvious difference between the pixel values of the face and the background in the 2D image, but there is a big difference between the face and the background in the depth image, therefore, the background part in the 2D image can be filtered out according to the depth information in the depth image, the interference of the background noise is reduced, and the accuracy of age prediction is improved. Multi-feature extraction is to use multiple convolutional neural networks to extract age, gender and race features. Multi-feature adaptive fusion is to automatically learn the weight parameters of the influence of age, gender and race features on age recognition based on the attention focusing mechanism, and then adaptively fuse these features according to the respective parameters for the final age prediction.

[0065] Age recognition is to predict the age information of the face in the image, which has important significance for the construction of a smart and safe city. Since most of the traditional age recognition methods are based on 2D images, the prediction result is easily affected by light and posture. In addition, the traditional method only extracts a single age feature, ignoring the influence of gender and race information on the age feature. Therefore, the present application proposes an age recognition method based on 3D face image, which extracts age, gender and race features from 3D face image (RGB-D), and proposes an adaptive feature fusion strategy to fuse these features into more stable and more discriminative age features, thereby reducing the influence of light, posture, gender and race on age prediction, and providing basic technical support for age recognition in complex scenes such as outdoors.

[0066] Innovatively, 3D face image is applied to age recognition. Since 3D image is not sensitive to changes in light and posture, the age features extracted from 3D image have strong robustness to changes in light and posture, reducing the error caused by changes in light and posture, and providing a basis for age recognition in complex scenes.

[0067] Multiple convolutional neural networks are used to extract age, gender and race features, and an adaptive feature fusion strategy is proposed to fuse these features into more robust age features, so that the model can adaptively adjust the parameters according to the differences in gender and race when predicting age, thereby improving the flexibility of the model.

[0068] A new background noise filtering method is proposed, the depth image information in the RGB-D image is used to calculate the range of face depth information, the background noise in the 2D image is filtered out according to the face depth range, so that only the face image area is reserved, the interference of noise is reduced, and the accuracy of age recognition is improved.

[0069] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation modes of the application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the implementation modes are not required or can not be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the claims of the application.

Claims

1. An age recognition method based on multi-feature fusion of 3D face images, characterized in that, Includes the following steps: S1: Filter out background noise in 2D images based on depth information in the depth image; S2: Use multiple convolutional neural networks to extract age, gender, and race features from 2D images after filtering out background noise; S3: Based on an attention-focusing mechanism, automatically learn the weight parameters of the influence of age, gender, and race characteristics on age recognition, and then fuse these features according to their respective parameters for age prediction; in step S3, These are age, gender, and race characteristics, among which The number of channels in the feature map. The size of the feature map; After passing through the CMP layer, it is converted to Then it is input into the fully connected layer and the activation function layer, and the output is... This is a one-dimensional column vector, representing the attention concentration distribution maps within the age, gender, and race features, respectively; that is, the weight values ​​of the feature maps for each channel. After connecting the channels, the fully connected layer and activation function layer are input to obtain the overall attention concentration distribution map, which is then combined with the original combined feature map. Perform convolution operations to obtain adaptive fused feature maps Adaptive fusion of feature maps Used for age recognition, it automatically adjusts the overall influence factors of age, gender, and race characteristics on age prediction results, so that the model makes weight adjustments based on the gender and race information of the face when predicting age.

2. The age recognition method based on multi-feature fusion of 3D face images according to claim 1, characterized in that, In step S1, It includes the following steps: S11: Use local binary pattern features to traverse the entire 2D image and detect the initial location of the face; S12: Crops out the corresponding face region in the initial area and calculates the histogram of pixel values ​​on the face region; S13: Obtain a face image with background noise filtered out based on the face depth information from the histogram.

3. The age recognition method based on multi-feature fusion of 3D face images according to claim 2, characterized in that, In step S11, the local binary pattern feature is obtained by traversing the entire image row by row and column by column using an LBP operator to calculate the feature map of the image.

4. The age recognition method based on multi-feature fusion of 3D face images according to claim 3, characterized in that, In step S12, after obtaining the face region, there is still background noise in the region. Based on the size of the area occupied by the face, the range of pixel values ​​is compared to obtain a histogram of pixel values.

5. The age recognition method based on multi-feature fusion of 3D face images according to claim 4, characterized in that, Depth images are captured using a Kinect camera, which is used to acquire depth information. The following formula can be used to filter out background noise: Iterate through each pixel in the face depth image. If the pixel value of that point is within... If the pixel value is outside the range, the pixel value of the corresponding point in the 2D image is set to 0; if the pixel value is within the range... Within the specified range, the pixel values ​​of corresponding points in the 2D image remain unchanged; after traversing each pixel, a face image with background noise filtered out can be obtained; where... It is the largest component in the histogram. It is the range length. The original pixel value. These are the pixel values ​​obtained after filtering.

6. The age recognition method based on multi-feature fusion of 3D face images according to claim 5, characterized in that, In step S2, after features are extracted by the convolutional layers of the convolutional neural network, a non-linear ReLU activation function is used to activate the extracted features. This activation function can suppress the negative output of the convolutional layer and amplify the positive output of the convolutional layer. After the activation function, a batch normalization layer is used to normalize the feature output of multiple samples. After the normalization layer, the features are then input into the pooling layer to extract the most representative features, while reducing the number of features and improving the calculation speed.

7. The age recognition method based on multi-feature fusion of 3D face images according to claim 6, characterized in that, Age, race, and gender features are obtained by training three networks using three different loss functions. Images are labeled with these three pieces of information and then fed into the networks for training, ultimately learning these features. In the gender convolutional network, the output is a two-bit one-dimensional vector, where [0,1] represents male and [1,0] represents female. In the race convolutional network, the output is a three-bit one-dimensional vector, where [0,0,1] represents Asian, [0,1,0] represents European, and [1,0,0] represents African. By continuously training the convolutional neural network, the gender and race features of the face are learned and stored in the convolutional neural network, thus extracting gender and race features.

8. The age recognition method based on multi-feature fusion of 3D face images according to claim 1, characterized in that, CMP is a channel maximum feature pooling layer used to select the maximum eigenvalue of each channel in the feature map.

Images