Feature fusion recognition method, device and storage medium for occluded faces

Abstract

The present application discloses a method, device and storage medium for feature fusion recognition of an occluded face. The present application relates to the field of image data processing technology. The method for feature fusion recognition of an occluded face includes: obtaining a facial region image; extracting a first facial feature of an unoccluded region in the facial region image based on an explicit feature enhancement module, and determining a second facial feature corresponding to the occluded region based on an implicit feature mining module; fusing the first facial feature and the second facial feature based on a feature fusion module to determine the fused facial feature corresponding to the facial region image; and outputting the recognition result of the fused facial feature. The present application can achieve the technical effect of realizing face recognition using a small amount of training data in a deep and complex network model, thereby improving the accuracy of face recognition.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a method, device and storage medium for feature fusion recognition of an occluded face. Background Art

[0002] Since convolutional deep networks have excellent performance in feature extraction, related technologies mainly use convolutional deep networks to extract features from facial images. The deeper the convolutional deep network is designed, the stronger the extracted features will be. Therefore, current convolutional neural network models are usually designed to be deeper to improve the accuracy of face recognition. However, deep and complex network models require a relatively large amount of data to avoid overfitting of the model, which makes the model's resource usage too high and its applicability on mobile devices poor, which in turn leads to low face recognition accuracy of the recognition model that can be loaded by current mobile devices. Summary of the invention

[0003] The main purpose of this application is to provide a feature fusion recognition method for occluded faces, aiming to solve the technical problem that the model in the related technology occupies too much resources and has poor applicability on mobile devices, which leads to low face recognition accuracy of the recognition model that can be loaded by current mobile devices.

[0004] To achieve the above-mentioned purpose, the present application proposes a feature fusion recognition method for occluded faces, which comprises: obtaining a facial area image; extracting a first facial feature of an unoccluded area in the facial area image based on an explicit feature enhancement module, and determining a second facial feature corresponding to the occluded area based on an implicit feature mining module; fusing the first facial feature and the second facial feature based on a feature fusion module to determine a fused facial feature corresponding to the facial area image; and outputting a recognition result of the fused facial feature.

[0005] In one embodiment, the explicit feature enhancement module is designed based on an attention mechanism, and the step of extracting the first facial feature of the unobstructed area in the facial area image based on the explicit feature enhancement module includes:

[0006] Determining an occlusion boundary of the facial region image based on the explicit feature enhancement module;

[0007] Determine the unoccluded area according to the occlusion boundary and the attention mechanism;

[0008] The first facial feature corresponding to the unobstructed area is extracted.

[0009] In one embodiment, the implicit feature mining module includes a discriminator and a generator, and the step of determining the second facial feature corresponding to the occluded area according to the implicit feature mining module includes:

[0010] Determining a completed image of the occluded area based on the generator and overall features of the facial area image;

[0011] Determining the credibility of the completed image based on the discriminator;

[0012] If the credibility does not meet the trust condition, updating the completed image;

[0013] The second facial feature corresponding to the completed image is extracted.

[0014] In one embodiment, the discriminator includes a local discriminator and a global discriminator, and the step of judging the credibility of the completed image based on the discriminator includes:

[0015] If the completed image is a partial image, determining the credibility based on internal features of the completed image by the partial discriminator;

[0016] If the completed image is a global image, determining the boundary features between the completed image and the unobstructed area by the global discriminator;

[0017] The credibility is determined according to the boundary feature.

[0018] In one embodiment, before the steps of extracting the first facial feature of the unobstructed area in the facial region image based on the explicit feature enhancement module and determining the second facial feature corresponding to the obstructed area based on the implicit feature mining module, the steps include:

[0019] Prune the residual network layer in the convolutional neural network and determine the target convolutional neural network;

[0020] Fine-tune the target convolutional neural network based on the weights before pruning as training parameters;

[0021] If the network accuracy of the fine-tuned target convolutional neural network is greater than or equal to a preset threshold, the residual network layer in the pruned convolutional neural network is executed based on the fine-tuned target convolutional neural network to determine the step of the target convolutional neural network.

[0022] In one embodiment, the step of pruning the residual network layer in the convolutional neural network and determining the target convolutional neural network includes:

[0023] Determining a prunable layer in the residual network layer;

[0024] Determining the importance index of the convolution filter in each of the prunable layers;

[0025] Determine the mean of the importance index corresponding to each of the prunable layers;

[0026] Determine the deviation between the importance index and the mean of the importance index in each of the prunable layers, and the neuron with a value greater than the mean of the deviation of the layer is the target neuron;

[0027] The target neuron is deleted to determine the target convolutional neural network.

[0028] In one embodiment, the step of fine-tuning the target convolutional neural network based on the weights before pruning as training parameters includes:

[0029] Inserting a recognition loss function into a network layer adjacent to the target convolutional neural network;

[0030] Determine the identification channel based on the recognition loss and reconstruction error of the corresponding feature map of the target convolutional neural network after insertion;

[0031] The target convolutional neural network is fine-tuned according to the identified channel.

[0032] In one embodiment, after the step of fine-tuning the target convolutional neural network based on the weights before pruning as training parameters, the following steps are included:

[0033] If the network accuracy of the fine-tuned target convolutional neural network is less than a preset threshold, the target convolutional neural network is compressed, wherein the loss function of the compression process includes at least one of instance loss and pairwise loss.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a feature fusion recognition device for an occluded face, the feature fusion recognition device for an occluded face comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the feature fusion recognition method for an occluded face as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and the computer-readable storage medium stores a program for implementing a feature fusion recognition method for an occluded face. The program for implementing the feature fusion recognition method for an occluded face is executed by a processor to implement the steps of the feature fusion recognition method for an occluded face as described above.

[0036] The present application provides a feature fusion recognition method for an occluded face. The present application first obtains a facial region image; extracts the first facial feature of the unoccluded region in the facial region image based on an explicit feature enhancement module, and determines the second facial feature corresponding to the occluded region based on an implicit feature mining module; fuses the first facial feature and the second facial feature based on a feature fusion module to determine the fused facial feature corresponding to the facial region image; and outputs the recognition result of the fused facial feature. The present application can effectively solve the technical problem that the model in the related technology occupies too many resources and has poor applicability on mobile devices, which leads to the low accuracy of face recognition of the recognition model that can be loaded by the current mobile device, and achieves the technical effect of improving the accuracy of face recognition of mobile terminals under low load conditions.

[0037] In summary, the present application extracts facial features of face images by combining an explicit feature enhancement module and an implicit feature mining module, simplifies the network model, reduces model parameters and computational complexity, and enables the deep network model to run better on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart diagram of the first embodiment of the feature fusion recognition method for occluded faces provided in this application;

[0041] Figure 2 A flowchart diagram of the second embodiment of the feature fusion recognition method for occluded faces provided in this application;

[0042] Figure 3 A flowchart diagram of the third embodiment of the feature fusion recognition method for occluded faces provided in this application;

[0043] Figure 4 A flowchart diagram of the fourth embodiment of the feature fusion recognition method for occluded faces provided in this application;

[0044] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the feature fusion recognition device for occluded faces in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The main solution of the embodiment of the present application is: obtaining a facial area image; extracting a first facial feature of an unobstructed area in the facial area image based on an explicit feature enhancement module, and determining a second facial feature corresponding to the obstructed area based on an implicit feature mining module; fusing the first facial feature and the second facial feature based on a feature fusion module to determine a fused facial feature corresponding to the facial area image; and outputting a recognition result of the fused facial feature.

[0049] Since related technologies utilize the important role of convolutional neural networks in feature extraction, when designing deep networks, the performance characteristics of convolutional neural networks become deeper as the network design deepens. However, this complex network model requires a relatively large amount of data to avoid overfitting of the model, and the training process will also bring a relatively large amount of computing and storage resources.

[0050] The present application provides a feature fusion recognition method for occluded faces, which can solve the problem that deep and complex network models require a relatively large amount of data to avoid overfitting of the model, resulting in excessive resource occupancy of the model and poor applicability on mobile devices, which in turn leads to the low face recognition accuracy of the recognition model that can be loaded by current mobile devices.

[0051] It should be noted that the execution subject of this embodiment can be a feature fusion recognition system for occluded faces, which is also a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes the feature fusion recognition system for occluded faces as an example to illustrate this embodiment and the following embodiments.

[0052] Based on this, the first embodiment of the present application provides a feature fusion recognition method for an occluded face, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the feature fusion recognition method for occluded faces in the present application.

[0053] In this embodiment, the feature fusion recognition method for occluded faces includes steps S100 to S400:

[0054] Step S100, obtaining a facial region image;

[0055] It should be noted that the face region image refers to the face region image located and extracted from the input image. It is understandable that the input image can be a photo or an image obtained by extracting a video frame from an input video, and the image acquisition source is not limited here.

[0056] An optional implementation manner uses a face detection algorithm to identify the approximate outline and position of a face, locates the face in an image, and extracts a facial area image by cropping the image.

[0057] For example, when taking a photo using a camera, the application first uses a built-in face detection algorithm to detect the face position in the photo, and then crops the image of the face area after performing an alignment operation.

[0058] Step S200, extracting a first facial feature of an unobstructed area in the facial region image based on an explicit feature enhancement module, and determining a second facial feature corresponding to the obstructed area based on an implicit feature mining module;

[0059] It should be noted that, in this embodiment, explicit features refer to features that can be directly observed from the image of the unobstructed face area image. Explicit features can be eyes, mouth, nose, facial contours, skin texture, etc. of a human face. The first facial feature refers to the feature extraction of the unobstructed area by the explicit feature enhancement module. Feature enhancement refers to amplifying important features that are not obvious, blurred or on the boundary to make the image recognition more prominent and easier to identify. Implicit features refer to features that are obscured, not directly visible or not easy to extract directly from the image. Implicit features can be the degree of fit between the face and the obstruction, changes in lighting and shadows, etc. The second facial feature refers to feature extraction of the obstructed area with the help of an implicit feature mining module. It can be understood that feature mining means that with the help of partially visible features, the complete features can be roughly inferred. For example, a photo of a person wearing a mask smiling is captured, and the features of the smile under the mask are inferred based on the degree of curvature of the eyes or changes in facial muscles.

[0060] An optional implementation method is to first analyze the facial area image after acquiring the facial area image using the display feature enhancement module, identify the unobstructed parts, and extract the features of the unobstructed areas; at the same time, the implicit feature mining module analyzes the same facial area image, uses a deep learning model to predict the features of the occluded areas, and generates a feature vector.

[0061] For example, when wearing a mask, the mouth and nose areas may be blocked. The display feature enhancement module extracts the first facial features based on the unblocked areas such as the eyes and forehead, and the implicit feature mining module attempts to predict the features of the mouth and nose, and extracts the implicit features in the unblocked areas as the second facial features.

[0062] Step S300, fusing the first facial feature and the second facial feature based on a feature fusion module to determine a fused facial feature corresponding to the facial region image;

[0063] It should be noted that the feature fusion module refers to combining explicit features and implicit features to form a more comprehensive and robust facial feature representation. The robustness refers to the ability of a system, model or function to maintain stability and performance in the face of erroneous input, work pressure or unexpected situations.

[0064] In an optional implementation, the explicit feature enhancement module is used to extract the first facial features that are directly visible in the unobstructed area, and the implicit feature mining module is used to analyze the potentially obstructed area to infer the second facial features that are not directly visible. Then, the two sets of features are integrated using the feature fusion module. It is feasible to achieve this through a weighted average fusion strategy to ensure that the final feature vector can represent the entire facial area.

[0065] For example, in an airport security check scenario, a passenger may cover his face with a mask. In this case, the explicit feature enhancement module extracts the first facial features of the unobstructed eye and forehead areas, while the implicit feature module tries to infer the second facial features obstructed by the mask. Subsequently, the feature fusion module combines these two sets of features to form a comprehensive facial feature representation.

[0066] Step S400: outputting the recognition result of the fused facial features.

[0067] It should be noted that after the fused facial features are obtained, these fused features are matched with the facial features stored in the database. The database here can be a local data warehouse or an external third-party comparison database, and there is no limitation on the database.

[0068] As an optional implementation, based on the facial features generated by the feature fusion module, the similarity between the face to be identified and each template in the database is first calculated, and the fused features are matched with the feature templates stored in the database. This can be achieved through an algorithm. The recognition system selects the most matching template as the recognition result and outputs the corresponding identity information.

[0069] For example, when an employee joins the company, they register their facial features, which are stored in a database. When an employee walks to the entrance of a sensitive area, the facial recognition system automatically captures their facial image, processes the image with the feature fusion module, extracts and fuses the facial features, and then compares the facial features with the templates in the database to determine the employee's identity.

[0070] In this embodiment, the image processing technology is used to accurately locate and obtain the facial area image, and then the explicit feature enhancement module is used to extract the obvious first facial features from the unobstructed area. At the same time, the implicit feature mining module is used to analyze and infer the possible second facial features of the occluded area. Finally, the two facial features are combined through the feature fusion module to form a comprehensive facial feature representation, thereby enhancing the system's ability to recognize faces under occluded conditions.

[0071] Based on the first embodiment, the second embodiment of the present application provides a feature fusion recognition method for an occluded face, referring to Figure 2 , Figure 2 This is a flow chart of the second embodiment of the feature fusion recognition method for occluded faces of the present application. Step S200 includes steps A210 to A230:

[0072] Step A210: determining an occlusion boundary of the facial region image based on the explicit feature enhancement module.

[0073] It should be noted that the occlusion boundary refers to the boundary of the area where the facial features are invisible due to certain objects or materials in the image. The occluders can be glasses, masks, hats, hair, or any other objects that block the face.

[0074] In an optional implementation, the explicit feature enhancement module analyzes the image to identify key feature points of the face, and using the key feature points as a reference, the module detects the boundaries of regions that do not match or are incomplete with known facial features, thereby obtaining occlusion boundaries of the image.

[0075] For example, in a facial recognition system for airport security, when a passenger's facial image is captured, the explicit feature enhancement module identifies key feature points such as the eyes, nose and mouth. For passengers wearing masks, the module detects that the mouth area is incomplete due to occlusion by the mask, and then uses image segmentation technology to determine the occlusion boundary of the mask.

[0076] Step A220, determining the unoccluded area according to the occlusion boundary and the attention mechanism.

[0077] It should be noted that the attention mechanism refers to a technique in deep learning that enables the model to focus on specific parts of the input data.

[0078] As an optional implementation manner, in this embodiment, after the occlusion boundary is determined, an attention mechanism is used to identify and focus on the unoccluded facial area, thereby determining the unoccluded area.

[0079] Step A230: extracting the first facial feature corresponding to the unblocked area.

[0080] It can be understood that after the unobstructed area is determined, the first facial feature of the unobstructed area is extracted based on the explicit feature enhancement module according to the image of the unobstructed area.

[0081] Based on any of the above embodiments, the third embodiment of the present application provides a feature fusion recognition method for an occluded face, referring to Figure 3 , Figure 3 This is a flow chart of the third embodiment of the feature fusion recognition method for occluded faces of the present application. Step S200 further includes steps A250 to A280:

[0082] Step A250, determining a completion image of the occluded area based on the generator and the overall features of the facial area image.

[0083] It should be noted that the generator network consists of an encoder, a converter, and a decoder. The encoder consists of 5 convolutional layers, the decoder consists of 5 convolutional layers, and the converter consists of 6 residual blocks. The generator here is to combine different channels of the encoder output image, and then convert the occluded image into an unoccluded image according to the characteristics of the image. The completed image is the image after the occluded area is completed by the generator.

[0084] Step A260: determining the credibility of the completed image based on the discriminator.

[0085] It should be noted that after the generator generates the completed image, its credibility needs to be judged to ensure that the completion result is both real and reliable. The discriminator is similar to the generator, which is also a model or network used to distinguish between two different types of inputs. At the same time, credibility refers to the visual fidelity of the completed image and the consistency of the original image content.

[0086] As an optional implementation, a dual discriminator is used to determine whether the locally generated partial image and the globally generated image are realistic enough. Since the discriminator is a binary model, in the initial stage of model training, different types of images input into the discriminator will cause instability. The cross entropy loss can be used to optimize the generator. The binary model refers to the use of simple decision rules to determine the authenticity of the image. Cross entropy loss is a commonly used classification loss function used to optimize the discriminator.

[0087] Further, step A260 includes steps A261 to A263:

[0088] Step A261: If the completed image is a partial image, the credibility is determined based on the internal features of the completed image by the local discriminator.

[0089] As an optional implementation, after determining the completed image, a local discriminator is used to use a pooling operation to reduce the dimension of the features converted into feature vectors, so as to determine the authenticity of the mask in the generated content. It should be noted that the pooling operation refers to the use of the pooling operation in the middle layer of the network, which helps to reduce the dimension of the features, thereby reducing the computational complexity to increase the processing speed of the network, and can extract the key features of the image and suppress unimportant noise.

[0090] Step A262: If the completed image is a global image, the boundary features between the completed image and the unobstructed area are determined by the global discriminator.

[0091] Step A263, determining the credibility based on the boundary feature.

[0092] As an optional implementation, after determining the completed image, a global discriminator is used to compare whether the newly generated content is authentic and consistent with the surrounding semantics, so as to judge the authenticity of the entire completed image. After the judgment is completed, the output is added to the global discriminator network to determine the credibility corresponding to the boundary features.

[0093] Step A270: if the credibility does not meet the trust condition, update the supplementary image.

[0094] Step A280: extracting the second facial feature corresponding to the completed image.

[0095] As an optional implementation, after determining the credibility, if the credibility of the completed image does not meet the condition, it means that the image features of the completed image deviate from the boundary features and need to be further optimized. It is understandable that after updating the completed image, the second facial features corresponding to the completed image are extracted.

[0096] In this embodiment, the occluded boundary of the facial area is determined based on the display feature enhancement module, and then the unoccluded area is determined according to the attention mechanism, so as to extract the first facial feature corresponding to the unoccluded area. At the same time, according to the completed image generated by the generator for the occluded area, the discriminator is used to judge the second facial feature, and the facial features are extracted from two angles respectively, thereby improving the performance and accuracy of the face recognition system.

[0097] Based on any of the above embodiments, the fourth embodiment of the present application provides a feature fusion recognition method for an occluded face, referring to Figure 4 , Figure 4This is a flowchart of the fourth embodiment of the feature fusion recognition method for occluded faces of the present application, and before step S200, steps A170 to A190 are also included:

[0098] Step A170, pruning the residual network layer in the convolutional neural network and determining the target convolutional neural network.

[0099] It should be noted that before obtaining the facial feature image of the image, the training model needs to be trained to ensure the model performance while reducing the complexity of the model.

[0100] As an optional implementation, first, the residual network is trained to ensure that the network can learn effective feature representations, and then the importance of each neuron in each layer is evaluated by a preset metric. Then, based on the importance score, neurons or weights that have little impact on network performance are selectively removed. After pruning, the network is fine-tuned to compensate for the performance loss caused by pruning. Finally, the target convolutional neural network is determined. It should be noted that the preset metric can be the absolute value of the weight, a specific eigenvalue of the Hessian matrix, etc., which is not limited here.

[0101] Further, step A170 includes steps A171 to A175:

[0102] Step A171, determining a prunable layer in the residual network layer;

[0103] Step A172, determining the importance index of the convolution filter in each of the prunable layers;

[0104] Step A173, determining the mean value of the importance index corresponding to each of the prunable layers;

[0105] Step A174, determining the deviation between the importance index in each of the prunable layers and the mean of the importance index, and the neuron with a deviation greater than the mean of the layer is the target neuron;

[0106] Step A175, delete the target neuron to determine the target convolutional neural network.

[0107] It should be noted that the prunable layer refers to the layer in this neural network, especially the residual network, in which some neurons or connections can be safely reduced without significantly affecting the network performance. Convolutional filters are the basic building blocks in convolutional neural networks and are used to extract features of input data.

[0108] As an optional implementation, based on the dimensional dependencies between some layers and layers in the previous residual block, the prunable layers in the residual network layer are determined, the prunable layers are colored yellow, and the non-prunable layers are colored red. Then, pruning is applied to a small number of layers in the model to determine the importance index of the convolution filter in each of the prunable layers. At the same time, the importance of any neuron in the prunable layer is calculated, and the average percentage of activation output (APoZ) of the neurons of all samples in the validation set is calculated. If the APoZ of the neuron in the prunable layer is greater than the standard deviation of the average APoZ in the same layer, the neuron and its connection are deleted to determine the target neuron.

[0109] Exemplarily, the steps for calculating the importance of convolutional filters in each layer are as follows: After pruning the zero-activated neurons, retrain the network using the weights before pruning as initialization values. Alternate pruning and retraining to further reduce zero activations in the network. Given the prunable layers are:

[0110] ,

[0111] Where n represents the total number of prunable layers in the network, and the importance of convolution filter c in layer i is calculated as follows:

[0112]

[0113] in and are the total number of validation samples and the dimension of the output feature map in the convolution filter c, respectively. f is calculated as follows:

[0114]

[0115] If the APoZ of a neuron is greater than the standard deviation of the mean APoZ in the same layer, the neuron and its connections are deleted.

[0116] Step A180, fine-tuning the target convolutional neural network based on the weights before pruning as training parameters.

[0117] It should be noted that once all layers are pruned, the network can be fine-tuned using the training parameters of the original data to restore the original accuracy as much as possible, and the training process is repeated until the network accuracy drops significantly after iterations. This process is stopped to avoid causing excessive disturbances to the pruned network structure.

[0118] Further, step A180 includes steps A181 to A183:

[0119] Step A181, inserting the recognition loss function into the network layer adjacent to the target convolutional neural network.

[0120] It should be noted that when training a convolutional neural network, the recognition loss function is inserted to improve the model performance and increase the recognition ability of the intermediate layer.

[0121] As an optional implementation, key layers in the network are selected, and then additional fully connected layers are inserted after these layers to act as micro-classifiers, which will calculate the loss of the difference between the output of these layers and the true label, i.e., the recognition loss.

[0122] Step A182, determining the identification channel based on the recognition loss and reconstruction error of the corresponding feature map of the target convolutional neural network after insertion.

[0123] It should be noted that determining the identification channel can improve network performance and efficiency, and it is necessary to insert the identification loss function and calculate the reconstruction error at the key layer of the network.

[0124] As an optional implementation, an additional loss function, such as classification loss, is inserted into the feature map of the target Convolutional Neural Networks (CNN), and the reconstruction error is calculated by comparing the difference between the original feature map and the feature map after network processing.

[0125] For example, insert P recognition losses Evenly insert into the network, represents the network layer with insertion loss, is the last layer. A loss can be added to each layer of the network, where , constructing a recognition loss , which uses The output of the layer is used as the input feature map. It can be expressed as:

[0126]

[0127] in, is the indicator function, represents the classifier weight of the fully connected layer, represents the number of input channels of the fully connected layer, and m is the number of classes. Reconstruction error It can be measured by the mean square error (MSE) between the baseline network feature map and the pruned network feature map, as shown below:

[0128]

[0129] in, Represents the feature map. Considering the discriminative perception loss and reconstruction error, the joint loss function is as follows:

[0130]

[0131] in Balancing the two terms, the algorithm flow of perceptual channel pruning is as follows:

[0132] First, initialize the model, through the pre-training model , training data { and parameters . Input the pruned model M.

[0133] insert Recognition loss } are inserted evenly into the network.

[0134] study , through and and With fine-tuning, the resolution of the middle layer can be significantly improved.

[0135] Initialize with M . Loop through each layer of perceptual pruning.

[0136] Step A183, fine-tuning the target convolutional neural network according to the identification channel.

[0137] It can be understood that after pruning each layer of neurons, fine-tuning the weights and parameters of the relevant channels further optimizes the network performance.

[0138] Step A190, if the network accuracy of the fine-tuned target convolutional neural network is greater than or equal to a preset threshold, the residual network layer in the pruned convolutional neural network is executed based on the fine-tuned target convolutional neural network to determine the step of the target convolutional neural network.

[0139] In an optional implementation, when determining the target convolutional neural network, the network accuracy of the target convolutional neural network is calculated. If the network accuracy is greater than or equal to a preset threshold, the target convolutional neural network is used for neural execution after fine-tuning.

[0140] Furthermore, step A180 also includes step B10:

[0141] Step B10: If the network accuracy of the target convolutional neural network after fine-tuning is less than a preset threshold, compress the target convolutional neural network, wherein the loss function of the compression process includes at least one of instance loss and pairwise loss.

[0142] It should be noted that with the help of knowledge distillation feature extraction method. The learning process consists of two stages. In the first stage, the input of the student model is initialized by the repair model. In the second stage, cross-quality knowledge extraction is performed, and the knowledge of ordinary faces contained in the teacher recognizer is transferred to the student recognizer by strengthening the consistency of the relationship structure. In this way, the student network for recognizing masked faces learns the representation of complete faces with the same clustering behavior as the original faces. Knowledge distillation is a widely used technique for transferring knowledge from a cumbersome teacher network to a compact teacher network. Traditional distillation usually focuses on classification tasks and is trained through cross entropy loss. During training, the output class distribution produced by the student model is forced to be close to the teacher model. In this way, the student model can achieve better results than directly accepting class label training. However, current distillation methods are still limited, and existing distillation methods usually focus on the point-by-point similarity between the representations of the teacher model and the student model. Instance relations can help reduce intra-class variations in the feature space and expand inter-class differences. However, this is rarely considered in distillation. It is then hypothesized that the constitutive knowledge is better represented by the relations of the learned representations than by the individuals of these representations, and that the structural distribution of unmasked faces can provide essential guidance for the identity feature rearrangement of finished faces. In addition, point-wise distillation methods usually require the teacher model and the student model to share similar network architectures and a close data domain.

[0143] Specifically, in the occluded face recognition scenario, we extract rich knowledge about the feature distribution of the unmasked face and use this rich knowledge to guide the face realignment. It is understandable that the common feature sought here is the aggregation behavior, that is, the instance relationship, which is more robust to network changes and domain shifts.

[0144] Let Ft(U)` and Fs( )` are respectively composed of the teacher network Ft(U) and the student network Fs( ), where U is the input of the teacher network. is the student network input. Ft(U)` is the feature extraction backend before the soFt(U)max layer, which is used to extract the identity features of the unmasked face. Fs( )` represents the layer before the embedding layer, which is used to extract the features of the mask. The training process of the student network can be described as transferring the relational structure Ft(U)` of the output representation to Fs( )`, to improve Fs( )`’s final recognition capability. and Respectively represent a set of n-order tuples, s i =Fs( )` is the knowledge that students gain from completing the face, t i=Ft(U)` is the teacher knowledge extracted from the corresponding basic knowledge. The loss function of the n-level distillation process can be expressed as:

[0145]

[0146] in is a relation potential function, n is used to measure the relation similarity between a given tuple teacher and student models, is a loss based on which structural differences are penalized. In order to effectively transfer relational knowledge, relational losses are introduced here in three orders, which implement structural similarity in an instance-wise, pair-wise, and triple-wise manner, respectively. The instance-based relational loss follows the common setting and enforces instance-based similarity by penalizing differences.

[0147]

[0148] choose Loss, not loss because it handles outliers better. In this task, in addition to the gap between the teacher and student domains, there are also large differences within the student domain. Despite better convergence and stronger robustness, the loss introduces unnecessary smoothness. Pairwise Relation Loss Several recent works have used pairwise relation distillation loss in tasks such as image classification, image retrieval, and semantic segmentation. It is used to transfer pairwise relations between instances, especially pairwise similarities in methods. It follows that the pairwise relation knowledge distillation loss is formulated as follows:

[0149]

[0150] in, is the Huber loss, is a pairwise potential function that measures the relationship between two instances in the mini-batch space. and The Euclidean distance between . is a regularizing factor that allows the relational structure to be transferred regardless of the spatial dimension difference between the source field and the task field. The triplet relation loss The structure in the triplet can provide stricter regularization than a pair. Inspired by this, the triplet relation distillation loss is proposed:

[0151]

[0152] in, is the Huber loss, and three instances are measured , and The corresponding triplet potential function for the angle formed in mini-batch space is formulated as:

[0153]

[0154] The triplet relation loss transfers the relations of instance embeddings by penalizing the angular difference. Compared with the pairwise potential function, the triplet potential function measures the structural similarity in a high-order space, thus achieving more effective relational knowledge transfer.

[0155] The total loss formula is as follows:

[0156]

[0157] in, is the cross entropy loss between the teacher and student network outputs. When cross entropy is used, the loss function is . , and are the various relevant distillation loss orders defined in Eqs. , and is a weighted hyperparameter to balance the loss term.

[0158] In this embodiment, unimportant layers or connections in the network are removed through a pruning strategy, aiming to reduce computational complexity and the number of parameters while maintaining the performance of the network as much as possible. Then, the target network is fine-tuned using the network weights before pruning as a starting point to restore or improve the performance that may be lost due to pruning. During the fine-tuning process, the network is re-acquired to a higher level of accuracy on the training data by carefully adjusting the weights. If the accuracy of the fine-tuned network meets or exceeds the preset performance threshold, this indicates that pruning and fine-tuning have successfully optimized the network while achieving the purpose of reducing computing resource consumption. This process not only improves the operating efficiency of the network, but also ensures that the network maintains sufficient accuracy and robustness in practical applications.

[0159] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the feature fusion recognition method for occluded faces in the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0160] The present application provides a feature fusion recognition device for an occluded face, the feature fusion recognition device for an occluded face comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the feature fusion recognition method for an occluded face in the above-mentioned embodiment 1.

[0161] Reference below Figure 5, which shows a schematic diagram of the structure of a feature fusion recognition device for occluded faces suitable for implementing the embodiments of the present application. The feature fusion recognition device for occluded faces in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The feature fusion recognition device for an occluded face shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0162] like Figure 5 As shown, the feature fusion recognition device for occluded faces may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the feature fusion recognition device for occluded faces are also stored in RAM1004. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the feature fusion recognition device for occluded faces to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a feature fusion recognition device for occluded faces with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0163] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0164] The feature fusion recognition device for occluded faces provided by the present application adopts the feature fusion recognition method for occluded faces in the above-mentioned embodiment, and can solve the technical problem of feature fusion recognition for occluded faces. Compared with the related art, the beneficial effects of the feature fusion recognition device for occluded faces provided by the present application are the same as the beneficial effects of the feature fusion recognition method for occluded faces provided by the above-mentioned embodiment, and the other technical features in the feature fusion recognition device for occluded faces are the same as the features disclosed in the method of the previous embodiment, and will not be repeated here.

[0165] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0166] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0167] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the feature fusion recognition method for occluded faces in the above-mentioned embodiment.

[0168] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0169] The computer-readable storage medium may be included in the feature fusion recognition device for occluding a face; or may exist independently without being assembled into the feature fusion recognition device for occluding a face.

[0170] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a feature fusion recognition device for an occluded face, the feature fusion recognition device for an occluded face: obtains a facial area image; extracts a first facial feature of an unoccluded area in the facial area image based on an explicit feature enhancement module, and determines a second facial feature corresponding to the occluded area based on an implicit feature mining module; fuses the first facial feature and the second facial feature based on a feature fusion module to determine a fused facial feature corresponding to the facial area image; and outputs the recognition result of the fused facial feature.

[0171] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0173] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0174] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned feature fusion recognition method for occluded faces, and can solve the technical problem of feature fusion recognition of occluded faces. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the feature fusion recognition method for occluded faces provided in the above-mentioned embodiments, and will not be elaborated here.

[0175] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the feature fusion recognition method for occluded faces as described above.

[0176] The computer program product provided by the present application can solve the technical problem of feature fusion recognition of occluded faces. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the feature fusion recognition method of occluded faces provided by the above embodiment, which will not be elaborated here.

[0177] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A feature fusion recognition method for occluded faces, characterized in that: The feature fusion recognition method for occluded faces comprises: Get the face area image; Prune the residual network layer in the convolutional neural network and determine the target convolutional neural network; The step of pruning the residual network layer in the convolutional neural network and determining the target convolutional neural network includes: Determining prunable layers in the residual network layer according to dimensional dependencies between some layers and layers in a previous residual block; Determine the importance index of the convolution filter in each of the prunable layers, wherein the importance index formula of the convolution filter c in layer i is as follows: in and are the total number and dimension of validation samples of the output feature map in the convolution filter c, respectively. For a prunable layer, f is calculated as follows: ； Determine the mean of the importance index corresponding to each of the prunable layers; Determine the deviation between the importance index and the mean of the importance index in each of the prunable layers, and the neuron with a value greater than the mean of the deviation of the layer is the target neuron; Deleting the target neuron to determine the target convolutional neural network; Fine-tune the target convolutional neural network based on the weights before pruning as training parameters; If the network accuracy of the fine-tuned target convolutional neural network is greater than or equal to a preset threshold, the residual network layer in the pruned convolutional neural network is executed based on the fine-tuned target convolutional neural network to determine the target convolutional neural network; If the network accuracy of the fine-tuned target convolutional neural network is less than a preset threshold, compressing the target convolutional neural network, wherein the loss function of the compression process includes at least one of an instance loss and a pairwise loss; Determining an occlusion boundary of the facial region image based on an explicit feature enhancement module, wherein the explicit feature enhancement module is designed based on an attention mechanism; Determine an unoccluded area according to the occlusion boundary and the attention mechanism; Extracting a first facial feature corresponding to the unobstructed area; Determine a second facial feature corresponding to the occluded area according to an implicit feature mining module, wherein the implicit feature mining module includes a discriminator and a generator, and the step of determining the second facial feature corresponding to the occluded area according to the implicit feature mining module includes: Determine the complement image of the occluded area based on the overall features of the generator and the facial area image, wherein the generator includes an encoder, a converter and a decoder, the encoder is composed of 5 convolutional layers, the decoder is composed of 5 convolutional layers, and the converter is composed of 6 residual blocks. Here, the generator is to combine different channels of the encoder output image, and then convert the occluded image into an unoccluded image according to the features of the image; Determining the credibility of the completed image based on the discriminator; If the credibility does not meet the trust condition, updating the completed image; Extracting the second facial feature corresponding to the completed image; fusing the first facial feature and the second facial feature based on a feature fusion module to determine a fused facial feature corresponding to the facial region image; Output the recognition result of the fused facial features.

2. The feature fusion recognition method for occluded faces according to claim 1, characterized in that: The discriminator includes a local discriminator and a global discriminator, and the step of judging the credibility of the completed image based on the discriminator includes: If the completed image is a partial image, determining the credibility based on internal features of the completed image by the partial discriminator; If the completed image is a global image, determining the boundary features between the completed image and the unobstructed area by the global discriminator; The credibility is determined according to the boundary feature.

3. The feature fusion recognition method for occluded faces according to claim 1, characterized in that: The step of fine-tuning the target convolutional neural network based on the weights before pruning as training parameters includes: Inserting a recognition loss function into a network layer adjacent to the target convolutional neural network; Determine the identification channel based on the recognition loss and reconstruction error of the corresponding feature map of the target convolutional neural network after insertion; The target convolutional neural network is fine-tuned according to the identified channel.

4. A feature fusion recognition device for an occluded face, characterized in that: The feature fusion recognition device for an occluded face includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the feature fusion recognition method for an occluded face as described in any one of claims 1 to 3.

5. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the feature fusion recognition method for an occluded face as described in any one of claims 1 to 3 are implemented.

Images