Fuzzy robust facial expression recognition method based on frequency domain adaptive weight adjustment
By introducing an adaptive fuzzy weight layer into the convolutional neural network and dynamically adjusting the weight of the frequency components, the problem of insufficient accuracy and robustness in the recognition of fuzzy images is solved, and higher recognition accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202411339068.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-02-28
Smart Images
Figure CN119541012A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a fuzzy robust facial expression recognition method based on frequency domain adaptive weight adjustment, belonging to the technical field of computer vision and pattern recognition. Background Art
[0002] With the development of deep learning and computer vision technology, facial expression recognition technology has been widely used in the fields of emotional computing, human-computer interaction, and intelligent monitoring. However, in actual application scenarios, image blur, such as turbulence blur caused by air flow or optical system distortion, or Gaussian blur caused by focus offset or camera shake during image acquisition, can significantly reduce the accuracy of expression recognition, posing a huge challenge to existing recognition algorithms. Current expression recognition methods mainly focus on processing in the spatial domain. Spatial domain methods can be divided into two categories: one is to restore the image through a deblurring algorithm before recognition, and the other is to directly extract blur-invariant features for recognition without deblurring. The former requires iterative calculation of the blur kernel, which is time-consuming and cannot meet real-time requirements; the latter, represented by local phase quantization (LPQ), is widely used in blur-invariant descriptors, which can extract blur-insensitive features of images without deblurring. However, spatial domain methods usually require complex models and high computing resources, and have limited effects when dealing with complex blurry scenes. In contrast, the frequency domain method converts the image from the spatial domain to the frequency domain through Fourier transform, so that information of different scales presents different distributions in the frequency domain, which is convenient for separating different components in the image. Moreover, the frequency domain method can enhance the key spectral components in the image that are not significantly affected by blurring without deblurring, thereby improving the recognition effect. However, the current traditional expression recognition methods are mostly focused on spatial domain processing, and the recognition effect of blurred images is poor, especially when dealing with complex blurred scenes, there are problems such as low recognition accuracy and poor robustness. Summary of the invention
[0003] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose a fuzzy robust facial expression recognition method based on a frequency domain convolutional neural network, which can cope with the challenge of fuzzy facial expression recognition in long-distance and real-time scenarios, and can improve the accuracy of facial expression recognition under fuzzy conditions. By analyzing the spectral characteristics of the image, the frequency components with less fuzzy influence are separated, thereby improving the network's recognition ability for fuzzy images.
[0004] The technical solution adopted by the present invention to solve the technical problem is: a fuzzy robust facial expression recognition method based on a frequency domain convolutional neural network, the method comprising the following steps:
[0005] Step 1: Calculate the difference in the spectrum matrix between the clear image and the blurred image through Fourier transform to quantify the effect of blur on different frequency components. The smaller the spectrum difference, the less sensitive the area is to blur; the larger the spectrum difference, the more significantly affected the area is.
[0006] Step 2: Based on the spectrum difference, an adaptive blur weighting layer (ABWL) is designed. The 48*48 spectrum image is used as the input of this layer and convolved with different filters. During the training process, the weights of each frequency component of the filter are dynamically adjusted through back propagation, so that the convolutional neural network can adaptively adjust the attention to different frequency components and enhance the recognition ability of blurred images.
[0007] Step 3: Design a Blur-Robust Frequency Convolutional Neural Network (BRF-CNN). This network model adds ABWL to the convolutional neural network to process the input spectrum image, highlight the blur-insensitive feature areas, then extract features from the processed image through CNN, and finally complete the classification through the fully connected layer.
[0008] Step 3-1: First, the 48*48 Fourier spectrum map is input to the ABWL layer. The filter of ABWL contains multiple masks of the same size of 48*48, which matches the size of the input image. Each input image is multiplied by a different filter mask to generate different versions of the image. These images are then concatenated together in the channel dimension to form the output data, and the shape becomes (48,48,3*n), where n represents the number of masks in the filter, which serves as the input of the convolutional layer.
[0009] Step 3-2: The spectrum map adjusted by the ABWL layer is input into the convolution layer for feature extraction. The convolution layer contains 64 filters, and the convolution kernel size is 3*3. After passing through the Conv2D layer, the shape of the feature map is (48,48,64), and then the image size is downsampled to (24,24,64) through maximum pooling. The downsampled feature map is flattened into a one-dimensional vector with a shape of (batch_size,36864).
[0010] Step 3-3: The one-dimensional feature vector passes through the fully connected layers of Dense(128) and Dense(7), and the final output layer has 7 nodes, corresponding to the 7 expression categories in expression recognition.
[0011] Experiments were conducted on the public CK+ dataset, and the results showed that BRF-CNN showed significant advantages over traditional spatial domain convolutional neural networks (CNN). In turbulent blur and Gaussian blur scenes, the average recognition accuracy reached 91.63% and 81.68%, respectively, which were 36.94 and 29.03 percentage points higher than traditional CNN, respectively, showing higher recognition accuracy and stronger robustness.
[0012] Furthermore, step 1 of the present invention comprises: first, the input blurred image I blur The spectrum representation is obtained through Fourier transform:
[0013]
[0014] in, represents Fourier transform, F(I blur ) is the representation of the blurred image in the frequency domain.
[0015] Next, by comparing the clear image I clear The difference between the spectrum of the clear image and the blurred image is used to quantify the effect of blur on different frequency components. clear ), the spectrum difference is expressed as:
[0016] AF=|F(I clear )-F(I blur )|
[0017] Based on this spectral difference, we can determine which frequency components are more affected by blurring and which are less affected.
[0018] Furthermore, step 2 of the present invention includes: the two-pole filter adopts a bandpass filter design, which is defined as follows:
[0019]
[0020] Among them, f bdy is the frequency of the filter low frequency and high frequency dividing line. In the experiment, five different dividing frequencies f are set bdy , respectively 0.9, 0.8, 0.7, 0.6 and 0.5, Figure 2 What is shown is that when f bdy Filter structure when =0.5.
[0021] Principle and design of circular filter:
[0022] like Figure 3 The circular filter shown is used to smooth a specific frequency range to enhance the performance of blur-insensitive areas. The principle is as follows:
[0023] First, a set of circular filters are designed with the center of the frequency domain as the center of the circle, which are defined as follows:
[0024]
[0025] Among them, r is the maximum radius of the filter. When r = 1, the filter generates a circular mask with a radius of 1; when r = 2, two masks are generated, corresponding to circular areas with radii of 1 and 2. Similarly, the larger r is, the more masks are generated. Figure 3 It shows that when the radius is 5, the group of filters contains five circular masks with different radii.
[0026] Furthermore, step 2 of the present invention includes: dynamically adjusting the weights of each frequency component of the filter by back propagation during the training process, so that the convolutional neural network can adaptively adjust the attention to different frequency components and enhance the recognition ability of blurred images. The weight update formula is:
[0027]
[0028] in, represents the loss function of the model, is the gradient of the loss function with respect to the weight w(f), and α is the learning rate.
[0029] Finally, the frequency component after weighted processing by the ABWL layer is expressed as:
[0030] F'(I blur )=w(f)·F(I blur )
[0031] Beneficial effects:
[0032] 1. The present invention can cope with the challenge of blurry facial expression recognition in long-distance and real-time scenarios, and can improve the accuracy of facial expression recognition under blurry conditions. By analyzing the spectral characteristics of the image, the frequency components with less blurring effects are separated, thereby improving the network's ability to recognize blurry images.
[0033] 2. The present invention can demonstrate higher recognition accuracy and stronger robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is an adaptive fuzzy weight layer (ABWL) designed by the present invention, which shows its working principle diagram of enhancing the robustness of the network to blurred images by dynamically adjusting the weights of frequency components.
[0035] Figure 2This is a schematic diagram of the two-pole filter mentioned in the present invention, taking a 5×5 two-pole filter as an example, showing how to selectively enhance or suppress different frequency components in the frequency domain.
[0036] Figure 3 This is a schematic diagram of the circular filter mentioned in the present invention. Taking a circular filter with a radius of 5 as an example, it shows the smoothing processing of a specific frequency range, which is used to distinguish the frequency components with greater blurring effects.
[0037] Figure 4 It is a structural diagram of the fuzzy robust frequency convolutional neural network (BRF-CNN) designed by the present invention, showing the design of each level of the network, including a schematic diagram of an adaptive fuzzy weight layer, a convolution layer, and a fully connected layer. DETAILED DESCRIPTION
[0038] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.
[0039] The present invention provides a fuzzy robust facial expression recognition method based on frequency domain convolutional neural network, the method comprising the following steps:
[0040] Step 1: Analyze the frequency spectrum difference between the clear image and the blurred image through Fourier transform to quantify the impact of blur on different frequency components.
[0041] Step 2: Based on the spectral difference, an adaptive blur weighting layer (ABWL) is designed. This layer dynamically adjusts the weights of each frequency component through a trainable filter, so that the convolutional neural network can adaptively adjust the attention to different frequency components and enhance the recognition ability of blurred images.
[0042] Step 3: Design a Blur-Robust Frequency Convolutional Neural Network (BRF-CNN) to analyze the spectrum differences and design different filters (two-pole filter and circular filter) to initialize the initial weights of ABWL.
[0043] Step 4: During the training process, the weights of the ABWL layer are optimized and adjusted through back propagation. The network can adaptively adjust the weights of different frequency components to enhance its robustness in blurred scenes.
[0044] Step 5: Experiments were conducted on the public CK+ dataset, and the results showed that BRF-CNN showed significant advantages over traditional spatial domain convolutional neural networks (CNNs). In turbulent blur and Gaussian blur scenarios, the average recognition accuracy reached 91.63% and 81.68%, respectively, which were 36.94 and 29.03 percentage points higher than traditional CNNs, respectively, showing higher recognition accuracy and stronger robustness.
[0045] The main technical solutions of the present invention include the following contents:
[0046] Principle and design of Adaptive Blur Weighting Layer (ABWL):
[0047] like Figure 1 As shown, the present invention can adaptively adjust the blur degree of different frequency components, so that the network can pay more attention to the frequency part that is not sensitive to blur, thereby improving the accuracy of facial expression recognition. Its basic principles are as follows:
[0048] First, the input blurred image I blur The spectrum representation is obtained through Fourier transform:
[0049]
[0050] in, represents Fourier transform, F(I blur ) is the representation of the blurred image in the frequency domain.
[0051] Next, by comparing the clear image I clear The difference between the spectrum of the clear image and the blurred image is used to quantify the effect of blur on different frequency components. clear ), the spectrum difference is expressed as:
[0052] AF=|F(I clear )-F(I blur )|
[0053] Based on this spectral difference, we can determine which frequency components are more affected by blurring and which are less affected.
[0054] Based on the spectrum difference, an initial weight is designed for each frequency component, with frequency components that are more affected by blurring being given smaller weights and frequency components that are less affected being given larger weights. The specific initial weight values depend on the design of the selected filter.
[0055] During model training, the weights of the ABWL layer can be dynamically adjusted through back propagation. The weight update formula is:
[0056]
[0057] in, represents the loss function of the model, is the gradient of the loss function with respect to the weight w(f), and α is the learning rate.
[0058] Finally, the frequency component after weighted processing by the ABWL layer is expressed as:
[0059] F′(I blur )=w(f)·F(I blur )
[0060] These weighted frequency components will be passed to the subsequent convolutional layer for feature extraction, thereby improving the expression recognition performance of blurred images.
[0061] Principle and design of two-pole filter:
[0062] like Figure 2 As shown, the present invention provides a two-pole filter, which can process low-frequency and high-frequency components in the frequency domain separately to cope with the different effects of blur on different frequency components.
[0063] The two-pole filter adopts a bandpass filtering design, which is defined as follows:
[0064]
[0065] Among them, f bdy is the frequency of the filter low frequency and high frequency dividing line. In the experiment, five different dividing frequencies f are set bdy , respectively 0.9, 0.8, 0.7, 0.6 and 0.5, Figure 2 What is shown is that when f bdy Filter structure when =0.5.
[0066] Principle and design of circular filter:
[0067] like Figure 3 The circular filter shown is used to smooth a specific frequency range to enhance the performance of blur-insensitive areas. The principle is as follows:
[0068] First, a set of circular filters are designed with the center of the frequency domain as the center of the circle, which are defined as follows:
[0069]
[0070] Among them, r is the maximum radius of the filter. When r = 1, the filter generates a circular mask with a radius of 1; when r = 2, two masks are generated, corresponding to circular areas with radii of 1 and 2. Similarly, the larger r is, the more masks are generated. Figure 3It shows that when the radius is 5, the group of filters contains five circular masks with different radii.
[0071] BRF-CNN network structure and learning process:
[0072] like Figure 4 As shown, the present invention proposes a BRF-CNN (Blur-Robust Frequency domain Convolutional Neural Network) network structure, which includes the following main modules:
[0073] Input layer: The input layer receives image data from the frequency domain. The size of the input frequency domain image is N×N, where N is the resolution of the image.
[0074] Adaptive Blur Weight Layer (ABWL): First, the input frequency domain image is processed by the adaptive blur weight layer. This layer dynamically adjusts its weights according to the degree to which different frequency components are affected by blur, so that the network can adaptively focus on those frequency components that are less affected by blur.
[0075] Convolutional layer: The image processed by the ABWL layer enters the convolutional layer. This layer extracts frequency features at different scales through convolution operations. The convolution kernel size is set to k×k (k=3 in this article), the stride is 1, and the ReLU activation function is used:
[0076]
[0077] Among them, w ij is the convolution kernel weight, and b is the bias term.
[0078] Fully connected layer: After feature extraction, the fully connected layer is used to map the extracted frequency domain features to expression categories. The input of the fully connected layer is the frequency domain feature vector, and the output is the classification result. The classification result is passed through the Softmax function to obtain the final expression category probability distribution:
[0079]
[0080] Among them, z i is the score corresponding to category i, and C is the total number of categories.
[0081] The present invention analyzes the spectral difference between blurred images and clear images, quantifies the impact of blur on the image spectrum, proposes an adaptive blur weight layer (ABWL), and uses a trainable filter to dynamically adjust the weights of different frequency components, so that BRF-CNN can adaptively adjust the focus on different frequency components, thereby enhancing the recognition ability and significantly improving the accuracy and robustness of facial expression recognition in blurry scenes.
[0082] In the simulation experiment, the present invention performs blur processing on the CK+ dataset image using the Gaussian degradation model:
[0083]
[0084] Generate a Gaussian blurred image; using the turbulence degradation model:
[0085]
[0086] Generates a turbulence blurred image.
[0087] Next, Fourier transforms are performed on the clear and blurred images to analyze their spectral differences and quantify the effect of blur on different frequency components. Through the adaptive blur weight layer (ABWL), trainable filters are used to dynamically adjust the network's response to frequency components that are less susceptible to blur, and the optimized frequency domain features are then input into the convolutional layer.
[0088] In the convolution part, the present invention uses 64 3×3 convolution kernels to extract local features, and performs feature downsampling through the maximum pooling layer. The pooled features are flattened and passed to the fully connected layer to generate a 128-dimensional feature vector. The network parameters are optimized through the back propagation algorithm, and finally high-precision expression recognition of blurred images is achieved, and 7-category classification prediction of the input image is performed through the output layer.
[0089] Experimental results show that the performance of the present invention on the CK+ dataset is significantly better than that of the traditional spatial domain method. In turbulence and Gaussian blur scenes, the average recognition accuracy of facial expressions at different blur levels reached 91.63% (compared to 54.69% of the spatial domain CNN model recognition rate) and 81.68% (compared to 52.65% of the spatial domain CNN model recognition rate), respectively, showing high recognition accuracy and strong robustness.
[0090] The technical parameters and simulation experiment settings of the present invention include the following:
[0091] Experimental environment: The environment and computer configuration are Intel Core i7 12700, 16G memory, NVIDIA3060 graphics card with 6G video memory, and the software platform is Python3.9, TensorFlow-gpu 2.10.0, NVIDIA CUDA 11.8.0, and cuDNN7.8.0 library.
[0092] Dataset: The most widely used laboratory control database CK+ in the field of facial expression recognition is used. It contains 536 image sequences collected from 123 subjects. The expressions of each sequence range from neutral expression to peak expression. 327 sequences are labeled into 7 typical expressions: anger, contempt, disgust, fear, happiness, sadness, and surprise. We select the last three frames of each sequence from these 327 labeled sequences, a total of 981 facial expression images, to construct training sets, validation sets, and test sets, divided into 7 / 2 / 1. Clear pictures are used for model training, and the trained model is directly applied to images with different blur levels for testing.
[0093] Network parameters: The initial momentum is 0.9, the initial learning rate is 0.01, and the ReduceLROnPlateau adaptive learning rate adjustment strategy is used. When the verification accuracy does not improve within 50 epochs, the learning rate is reduced by 0.75 to ensure the stability and convergence of the training. During the training process, the performance of the verification set is monitored, and the stochastic gradient descent (SGD) optimizer is used to update and optimize the network parameters.
[0094] Blur parameters: In the Gaussian blur scene, the blur kernel is set to 3, 5, 7, 9, and 11; in the turbulence blur scene, the turbulence effect with different blur levels is simulated, and the turbulence parameter k is set to 0.01, 0.02, 0.03, 0.04, and 0.05.
[0095] Evaluation results: The average recognition accuracy of the model in the two blurry scenes is 91.63% (turbulent blur) and 81.68% (Gaussian blur).
[0096] The invention is not limited to the above-mentioned embodiments and claims, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the invention.
Claims
1. A fuzzy robust facial expression recognition method based on frequency domain convolutional neural network, characterized in that: The method comprises the following steps: Step 1: Calculate the difference in the spectrum matrix between the clear image and the blurred image through Fourier transform to quantify the effect of blur on different frequency components. The smaller the spectrum difference, the less sensitive the area is to blur. The larger the spectrum difference, the more significantly affected the area is. Step 2: Based on the spectrum difference, an adaptive blur weighting layer (ABWL) is designed. The 48*48 spectrum image is used as the input of this layer and convolved with different filters. During the training process, the weights of each frequency component of the filter are dynamically adjusted through back propagation, so that the convolutional neural network can adaptively adjust the attention to different frequency components and enhance the recognition ability of blurred images. Step 3: Design a Blur-Robust Frequency Convolutional Neural Network (BRF-CNN), add ABWL to the convolutional neural network to process the input spectrum image, highlight the blur-insensitive feature area, then extract features from the processed image through CNN, and finally complete the classification through the fully connected layer; Step 3-1: First, the 48*48 Fourier spectrum map is input to the ABWL layer. The ABWL filter contains multiple masks of the same size of 48*48, which matches the size of the input image. Each input image is multiplied by a different filter mask to generate different versions of the image. These images are then concatenated together in the channel dimension to form output data with a shape of (48,48,3*n), where n represents the number of masks in the filter, which is used as the input of the convolution layer. Step 3-2: The spectrum map adjusted by the ABWL layer is input to the convolution layer for feature extraction. The convolution layer contains 64 filters, and the convolution kernel size is 3*3. After passing through the Conv2D layer, the shape of the feature map is (48,48,64), and then the image size is downsampled to (24,24,64) through the maximum pooling. The downsampled feature map is flattened into a one-dimensional vector with a shape of (batch_size,36864); Step 3-3: The one-dimensional feature vector passes through the fully connected layers of Dense(128) and Dense(7), and the final output layer has 7 nodes, corresponding to the 7 expression categories in expression recognition.
2. The fuzzy robust facial expression recognition method based on frequency domain convolutional neural network according to claim 1 is characterized in that: The step 1 comprises: First, the input blurred image I blur The spectrum representation is obtained by Fourier transform: in, represents Fourier transform, F(I blur ) is the representation of the blurred image in the frequency domain, Next, by comparing the clear image I clear The spectrum difference between the clear image and the blurred image is used to quantify the effect of blur on different frequency components. Let the spectrum of the clear image be F(I clear ), the spectrum difference is expressed as: ΔF=|F(I clear )-F(I blur )| Based on this spectral difference, it is possible to determine which frequency components are more affected by blurring and which are less affected.
3. The fuzzy robust facial expression recognition method based on frequency domain convolutional neural network according to claim 1, characterized in that: The step 2 includes: the two-pole filter adopts a bandpass filter design, which is defined as follows: Among them, f bdy Five different boundary frequencies f are set for the low-frequency and high-frequency boundary of the filter. bdy , respectively 0.9, 0.8, 0.7, 0.6 and 0.5; The circular filter is used to smooth a specific frequency range to enhance the performance of blur-insensitive areas. The principle is as follows: First, a set of circular filters are designed with the center of the frequency domain as the center of the circle, which are defined as follows: Among them, r is the maximum radius of the filter. When r=1, the filter generates a circular mask with a radius of 1; when r=2, two masks are generated, corresponding to circular areas with radii of 1 and 2 respectively, and so on. The larger r is, the more masks are generated.
4. The fuzzy robust facial expression recognition method based on frequency domain convolutional neural network according to claim 1, characterized in that: The step 2 includes: dynamically adjusting the weights of each frequency component of the filter by back propagation during the training process, so that the convolutional neural network can adaptively adjust the attention to different frequency components and enhance the recognition ability of blurred images. The weight update formula is: in, represents the loss function of the model, is the gradient of the loss function with respect to the weight w(f), α is the learning rate, Finally, the frequency component after weighted processing by the ABWL layer is expressed as: F′(I blur )=w(f)·F(I blur )。