Glasses removal method, device and communication equipment

Abstract

The present invention provides a glasses removal method, device and communication equipment, the method comprising: dividing an initial face image into a first image block, a second image block and a third image block; performing upsampling operations and downsampling operations on the second image block in sequence through a glasses removal shallow convolutional neural network, and reconstructing the image through convolution and pooling to obtain a reconstructed glasses-free image block with a size consistent with the second image block; splicing the reconstructed glasses-free image block, the first image block and the third image block to obtain a complete glasses-free face image; in the embodiment of the present invention, a higher-resolution feature map is first obtained through upsampling to improve the quality of the reconstructed image; then, feature extraction and reconstruction are performed on the feature map through operations such as pooling and a two-branch convolution module, and the process is also a downsampling process, and the size of the feature map is reduced to the size of the original image after the above processing; the neural network has fewer layers, thereby maintaining a smaller amount of calculation and a faster forward running efficiency.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a glasses removal method, device and communication equipment. Background Art

[0002] Glasses, one of the most common occluders in facial images, can interfere with facial feature extraction, reducing face recognition accuracy. Existing methods for removing glasses from facial images can be broadly categorized by the techniques used: traditional methods based on principal component analysis (PCA) and methods based on convolutional neural networks (CNNs). PCA-based methods were proposed earlier. Their principle is to identify the most important components in the data through PCA, remove redundancy and noise in the high-dimensional vector space, and use these principal components to represent the original data. In facial images reconstructed using PCA, the reconstruction error caused by glasses is dispersed throughout the entire reconstructed face image, resulting in noticeable eyeglasses traces in the reconstructed face image without glasses and a significant loss of image detail. Another feasible approach for removing glasses is to construct a suitable convolutional neural network and train it end-to-end. During the training process, backpropagation is used to fit the mapping between images of faces with and without glasses, thereby automatically removing glasses from facial images. The facial image reconstructed based on the CNN method is significantly improved compared with the traditional method. The difficulty of this method lies in constructing a suitable convolutional neural network structure so that it can accurately reconstruct facial images without glasses while being relatively lightweight and efficient.

[0003] The network model used in the glasses removal method in the prior art is a glasses removal deep convolutional neural network model. The shortcomings of this model include:

[0004] 1) To reduce the network's computational complexity, the model uses a downsampling-then-upsampling process: An image of a face wearing glasses is input. After a series of feature extraction operations, such as convolution and pooling, the resolution of the feature map decreases. To output a face image of the same size as the original, upsampling is used to enlarge the reduced feature map. Finally, the reconstructed image is mapped back to the original pixel space, and the output is a face image with the glasses removed. Because some feature information is lost during the downsampling process, the network model trained using this method produces lower-quality reconstructed images.

[0005] 2) This convolutional neural network model is a deep network, aiming to improve the network's feature extraction capabilities by increasing the number of network layers. As the number of network layers increases, the number of network parameters and computational complexity also increase, resulting in larger memory usage and longer forward execution times. Summary of the Invention

[0006] The purpose of the embodiments of the present invention is to provide a glasses removal method, apparatus, and communication device to solve the problems in the prior art of glasses removal deep convolutional neural network models, such as low image quality output, large network computational complexity, and long forward running time.

[0007] In order to solve the above problem, an embodiment of the present invention provides a method for removing glasses, comprising:

[0008] The initial face image is divided into a first image block, a second image block and a third image block, wherein the second image block includes a complete eyeglass portion;

[0009] Using the second image block as input to a glasses removal shallow convolutional neural network, obtaining a reconstructed glasses-free image block that is the same size as the second image block and output by the glasses removal shallow convolutional neural network; wherein the glasses removal shallow convolutional neural network is used to sequentially perform upsampling and downsampling operations on the second image block, and reconstruct the image through convolution and pooling to obtain the reconstructed glasses-free image block that is the same size as the second image block;

[0010] The reconstructed image block without glasses, the first image block and the third image block are stitched together to obtain a complete face image without glasses.

[0011] The glasses remove the processing step of the shallow convolutional neural network on the second image block, including:

[0012] performing an upsampling operation on the second image block using a deconvolution, a bilinear interpolation algorithm, or a super-resolution algorithm to obtain a high-resolution first feature map;

[0013] Reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having the same size as the second image block through downsampling;

[0014] The second feature map is mapped to the original pixel space using a target convolution layer to obtain the reconstructed glasses-free image block.

[0015] The reconstructing of the first feature map by convolution and pooling, and obtaining a second feature map having a size consistent with the second image block by downsampling operation, includes:

[0016] Remove the glasses frames whose brightness is lower than the skin in the first feature map through the maximum pooling operation of the filter, and fill the glasses occluded area with the surrounding skin pixels with higher brightness to obtain a third feature map;

[0017] Performing convolution processing on the third feature map through a dual-branch convolution module to obtain a fourth feature map and a fifth feature map;

[0018] Combining the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0019] The eyeglasses region having a brightness higher than that of the skin in the sixth feature map is removed by an average pooling operation of a filter to obtain the second feature map.

[0020] The step of performing convolution processing on the third feature map by a dual-branch convolution module to obtain a fourth feature map and a fifth feature map includes:

[0021] Dividing the third feature map into two groups of feature maps along the channel direction;

[0022] Inputting a set of feature maps into the first convolutional layer of the dual-branch convolutional module, and outputting the fourth feature map through the first convolutional layer;

[0023] Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

[0024] Wherein, the size of the initial face image is 128×128;

[0025] The size of the first image block is 128×10, the size of the second image block is 128×64, and the size of the third image block is 128×54.

[0026] The size of the first feature map is 256×128, and the number of output channels is 128.

[0027] The size of the maximum pooling operation is 2×2, and the step size is 2; the size of the third feature map is 128×64, and the number of output channels is 128;

[0028] The size of the average pooling operation is 3×3, the stride is 1, and the padding is 1; the size of the second feature map is 128×64, and the number of output channels is 128.

[0029] The convolution kernel size of the first convolutional layer is 3×3, the stride is 1, and the padding is 1; the size of the fourth feature map is 128×64, and the number of output channels is 64;

[0030] The convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the feature map output by the second convolution layer is 128×64, and the number of output channels is 128; the convolution kernel size of the third convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fifth feature map is 128×64, and the number of output channels is 64.

[0031] Wherein, the initial face image is a face image detected in face recognition detection, and the method further includes:

[0032] The complete face image without glasses obtained by stitching is used for face matching and recognition.

[0033] An embodiment of the present invention further provides a glasses removal device, comprising:

[0034] a segmentation module, configured to segment the initial face image into a first image block, a second image block, and a third image block, wherein the second image block includes a complete eyeglass portion;

[0035] a removal module, configured to use the second image block as input to a glasses removal shallow convolutional neural network, and obtain a reconstructed glasses-free image block of the same size as the second image block, output by the glasses removal shallow convolutional neural network; wherein the glasses removal shallow convolutional neural network is configured to sequentially perform upsampling and downsampling operations on the second image block, and reconstruct the image through convolution and pooling to obtain the reconstructed glasses-free image block of the same size as the second image block;

[0036] A stitching module is used to stitch the reconstructed image block without glasses, the first image block and the third image block to obtain a complete face image without glasses.

[0037] An embodiment of the present invention further provides a communication device, including a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, and the processor is configured to perform the following operations:

[0038] The initial face image is divided into a first image block, a second image block and a third image block, wherein the second image block includes a complete eyeglass portion;

[0039] Using the second image block as input to a glasses removal shallow convolutional neural network, obtaining a reconstructed glasses-free image block that is the same size as the second image block and output by the glasses removal shallow convolutional neural network; wherein the glasses removal shallow convolutional neural network is used to sequentially perform upsampling and downsampling operations on the second image block, and reconstruct the image through convolution and pooling to obtain the reconstructed glasses-free image block that is the same size as the second image block;

[0040] The reconstructed image block without glasses, the first image block and the third image block are stitched together to obtain a complete face image without glasses.

[0041] The processor is further configured to execute the following process:

[0042] performing an upsampling operation on the second image block using a deconvolution, a bilinear interpolation algorithm, or a super-resolution algorithm to obtain a high-resolution first feature map;

[0043] Reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having the same size as the second image block through downsampling;

[0044] The second feature map is mapped to the original pixel space using a target convolution layer to obtain the reconstructed glasses-free image block.

[0045] The processor is further configured to execute the following process:

[0046] Remove the glasses frames whose brightness is lower than the skin in the first feature map through the maximum pooling operation of the filter, and fill the glasses occluded area with the surrounding skin pixels with higher brightness to obtain a third feature map;

[0047] Performing convolution processing on the third feature map through a dual-branch convolution module to obtain a fourth feature map and a fifth feature map;

[0048] Combining the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0049] The eyeglasses region having a brightness higher than that of the skin in the sixth feature map is removed by an average pooling operation of a filter to obtain the second feature map.

[0050] The processor is further configured to execute the following process:

[0051] Dividing the third feature map into two groups of feature maps along the channel direction;

[0052] Inputting a set of feature maps into the first convolutional layer of the dual-branch convolutional module, and outputting the fourth feature map through the first convolutional layer;

[0053] Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

[0054] An embodiment of the present invention further provides a communication device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the glasses removal method described above when executing the program.

[0055] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned glasses removal method when executed by a processor.

[0056] The above technical solution of the present invention has at least the following beneficial effects:

[0057] In the glasses removal method, device and communication equipment of the embodiments of the present invention, a glasses removal shallow convolutional neural network is designed. In the glasses removal shallow convolutional neural network, a higher resolution feature map is first obtained by upsampling to improve the quality of the reconstructed image; then, the feature map is extracted and reconstructed through operations such as pooling and a two-branch convolution module. This process is also a downsampling process. After the above processing, the size of the feature map is reduced to the size of the original image; finally, the feature map is mapped to the original pixel space through convolution; the number of neural network layers is relatively small, thereby maintaining a small amount of computation and a faster forward operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart showing the steps of the glasses removal method provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram showing a glasses removal shallow convolutional neural network in an example provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram showing the structure of a glasses removal device provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram showing the structure of a communication device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0063] like Figure 1 As shown, an embodiment of the present invention provides a method for removing glasses, comprising:

[0064] Step 101: dividing an initial face image into a first image block, a second image block, and a third image block, wherein the second image block includes a complete eyeglass portion;

[0065] In this step, data preprocessing is performed to convert the initial face image into the first image block, the second image block, and the third image block. Facial key points are aligned to ensure that the second image block contains complete glasses information.

[0066] Step 102: Using the second image block as input to a glasses removal shallow convolutional neural network, obtaining a reconstructed glasses-free image block that is the same size as the second image block and output by the glasses removal shallow convolutional neural network. The glasses removal shallow convolutional neural network is used to sequentially perform upsampling and downsampling operations on the second image block, and reconstruct the image through convolution and pooling to obtain the reconstructed glasses-free image block that is the same size as the second image block.

[0067] This step first obtains a higher-resolution feature map through upsampling, then performs further feature extraction and image reconstruction through operations such as convolution and pooling, and downsamples the feature map to the image size of the second image block. Finally, convolution is used to map the feature map to the RGB color space (i.e., the original pixel space) and output the reconstructed non-glasses image block.

[0068] Step 103 : combining the reconstructed image block without glasses, the first image block, and the third image block to obtain a complete face image without glasses.

[0069] In the embodiment of the present invention, the glasses remove the training strategy of the shallow convolutional neural network, and its specific parameter settings can be adjusted according to the training situation, which will not be described in detail here.

[0070] As an optional embodiment, the size of the initial face image is 128×128; the size of the first image block is 128×10, the size of the second image block is 128×64; and the size of the third image block is 128×54.

[0071] In at least one embodiment of the present invention, the step of removing the processing of the second image block by the shallow convolutional neural network includes:

[0072] performing an upsampling operation on the second image block using a deconvolution, a bilinear interpolation algorithm, or a super-resolution algorithm to obtain a high-resolution first feature map;

[0073] Reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having the same size as the second image block through downsampling;

[0074] The second feature map is mapped to the original pixel space using a target convolution layer to obtain the reconstructed glasses-free image block.

[0075] In the embodiments of the present invention, upsampling methods include deconvolution, bilinear interpolation, and super-resolution algorithms. Bilinear interpolation is faster, but the output image quality is inferior to deconvolution. Super-resolution algorithms can achieve better results, but are computationally intensive, resulting in longer network runtimes than deconvolution. In practical applications, the appropriate upsampling method can be selected based on specific needs and is not specifically limited here.

[0076] As an optional embodiment, the size of the first feature map is 256×128, and the number of output channels is 128.

[0077] As an optional embodiment, reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having a size consistent with the second image block through a downsampling operation, includes:

[0078] Remove the glasses frames whose brightness is lower than the skin in the first feature map through the maximum pooling operation of the filter, and fill the glasses occluded area with the surrounding skin pixels with higher brightness to obtain a third feature map;

[0079] Performing convolution processing on the third feature map through a dual-branch convolution module to obtain a fourth feature map and a fifth feature map;

[0080] Combining the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0081] The eyeglasses region having a brightness higher than that of the skin in the sixth feature map is removed by an average pooling operation of a filter to obtain the second feature map.

[0082] Optionally, the size of the maximum pooling operation is 2×2, and the step size is 2; the size of the third feature map is 128×64, and the number of output channels is 128;

[0083] The size of the average pooling operation is 3×3, the stride is 1, and the padding is 1; the size of the second feature map is 128×64, and the number of output channels is 128.

[0084] In at least one optional embodiment of the present invention, performing convolution processing on the third feature map by a dual-branch convolution module to obtain a fourth feature map and a fifth feature map includes:

[0085] Dividing the third feature map into two groups of feature maps along the channel direction;

[0086] Inputting a set of feature maps into the first convolutional layer of the dual-branch convolutional module, and outputting the fourth feature map through the first convolutional layer;

[0087] Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

[0088] Optionally, the convolution kernel size of the first convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fourth feature map is 128×64, and the number of output channels is 64;

[0089] The convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the feature map output by the second convolution layer is 128×64, and the number of output channels is 128; the convolution kernel size of the third convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fifth feature map is 128×64, and the number of output channels is 64.

[0090] As another optional embodiment, the glasses removal method provided in the embodiment of the present invention can be applied to the face recognition process as the last step of face image preprocessing, that is, the initial face image is the face image detected in the face recognition detection, and the method further includes:

[0091] The complete face image without glasses obtained by stitching is used for face matching and recognition.

[0092] Because existing methods use deep models that take a long time to run, most existing face recognition methods based on glasses removal first determine whether glasses are present in the input image before performing the glasses removal process. A pre-trained detection model is then used to detect the location of glasses in the image. If glasses are detected, they are removed; otherwise, face recognition is performed directly. However, the shallow convolutional neural network proposed in the glasses removal method provided in the embodiments of the present invention can automatically remove glasses from images of faces wearing glasses while automatically retaining the features of the original images of faces without glasses. Furthermore, the shallow convolutional neural network has fewer layers and can run forward more quickly. Therefore, the glasses removal method provided in the embodiments of the present invention can be used directly to pre-process images for glasses removal without having to detect and determine whether glasses are present in the image.

[0093] like Figure 2 Shown is an example of a glasses removal method provided by an embodiment of the present invention and its corresponding glasses removal shallow convolutional neural network.

[0094] For the initial facial image, facial landmarks are aligned and the image is resized to 128×128. The 128×128 input image is decomposed into three image blocks of 128×10, 128×64, and 128×54 sizes, designated Part 1 (first image block), Part 2 (second image block), and Part 3 (third image block). Part 2 contains the complete eyewear information.

[0095] Deconvolution (DeConv1) is then used to upsample Part 2, outputting a 256×128 feature map with 128 output channels. Next, a max pooling operation with a filter size of 2×2 and a stride of 2 is used to remove glasses frames that are lower in brightness than the skin in the face image, and fill in the areas obscured by the glasses with the surrounding, brighter skin pixels. After max pooling, the width and height of the feature map are reduced to 128×64, while the number of output channels remains at 128.

[0096] The output feature maps are evenly divided into two groups along the channel direction and input into the two-branch convolution module. One group of feature maps is input into a convolution layer Conv2_1 with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, outputting a feature map of size 128×64 with 64 channels. The other group is sequentially passed through a convolution layer Conv3_1 with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, outputting a feature map of size 128×64 with 128 channels. It is then passed through a convolution layer Conv3_2 with a convolution kernel size of 3×3, a stride of 1, and a padding of 1, outputting a feature map of size 128×64 with 64 channels. In other words, the two-branch convolution module outputs two groups of feature maps, both of which are 128×64 in size and have 64 channels.

[0097] The two feature maps output by the dual-branch convolutional module are concatenated. Average pooling (Ave Pool) with a filter size of 3×3, a stride of 1, and a padding of 1 is then used to remove areas of the glasses that are brighter than the skin, such as the reflective areas of the glasses. After average pooling, the feature map size remains unchanged at 128×64, with 128 channels. Finally, a convolutional layer (Conv4) is used to map the feature map to RGB space, outputting a 3-channel reconstructed image patch of the same size as in Part 2. This reconstructed image patch is then combined with Parts 1 and 3 to produce the final, complete face image without glasses.

[0098] The present invention proposes a method for automatically removing glasses from facial images using a shallow convolutional neural network. This method uses upsampling followed by downsampling, first increasing feature resolution through deconvolution. Further feature extraction and image reconstruction are then performed sequentially through convolution, maximum pooling, a two-branch convolution module, and average pooling. The feature map is then downsampled to the input image size, and finally, convolution is used to map the feature map to the RGB color space. By processing the higher-resolution feature map, this method can produce higher-quality reconstructed facial images without glasses. Furthermore, the new network proposed in this embodiment has fewer layers. While existing methods use networks with depths exceeding 10 layers, resulting in a high number of parameters and computational complexity, the network proposed in this embodiment has only four layers and a total of five convolutions, significantly reducing the number of network layers compared to existing methods. Therefore, while employing an upsampling-then-downsampling process, the present invention can still maintain a low computational load and high forward efficiency.

[0099] In summary, the embodiments of the present invention, on the one hand, first obtain a higher-resolution feature map through upsampling, thereby improving the quality of the reconstructed image; then, downsampling is performed, and the image is reconstructed through operations such as convolution, pooling, and merging. Finally, convolution is used to map the feature map to the original pixel space. This results in fewer network layers, thereby maintaining a low computational load and faster forward operation efficiency. On the other hand, a trained convolutional neural network model is used to process facial images, outputting a reconstructed image of the face without glasses, thereby improving the facial recognition rate.

[0100] like Figure 3 As shown, an embodiment of the present invention further provides a glasses removal device, comprising:

[0101] A segmentation module 301 is configured to segment the initial face image into a first image block, a second image block, and a third image block, wherein the second image block includes a complete eyeglass portion;

[0102] a removal module 302 configured to use the second image block as input to a glasses removal shallow convolutional neural network, and obtain a reconstructed glasses-free image block of the same size as the second image block, output by the glasses removal shallow convolutional neural network; wherein the glasses removal shallow convolutional neural network is configured to sequentially perform upsampling and downsampling operations on the second image block, and reconstruct the image through convolution and pooling to obtain the reconstructed glasses-free image block of the same size as the second image block;

[0103] The stitching module 303 is configured to stitch the reconstructed face image without glasses, the first image block, and the third image block to obtain a complete face image without glasses.

[0104] As an optional embodiment, the removal module includes:

[0105] A first submodule is configured to perform an upsampling operation on the second image block by using a deconvolution, a bilinear interpolation algorithm, or a super-resolution algorithm to obtain a high-resolution first feature map;

[0106] A second submodule is configured to reconstruct the first feature map through convolution and pooling, and obtain a second feature map having the same size as the second image block through a downsampling operation;

[0107] The third submodule is used to use the target convolution layer to map the second feature map to the original pixel space to obtain the reconstructed glasses-free image block.

[0108] As an optional embodiment, the second submodule includes:

[0109] The first unit is configured to remove the glasses frames whose brightness is lower than that of the skin in the first feature map by performing a maximum pooling operation of a filter, and fill the glasses occluded area with surrounding skin pixels of higher brightness to obtain a third feature map;

[0110] A second unit is configured to perform convolution processing on the third feature map through a dual-branch convolution module to obtain a fourth feature map and a fifth feature map;

[0111] A third unit is configured to combine the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0112] The fourth unit is configured to remove the eyeglasses region having a brightness higher than that of the skin in the sixth feature map by performing an average pooling operation of a filter to obtain the second feature map.

[0113] As an optional embodiment, the second unit is further configured to:

[0114] Dividing the third feature map into two groups of feature maps along the channel direction;

[0115] Inputting a set of feature maps into the first convolutional layer of the dual-branch convolutional module, and outputting the fourth feature map through the first convolutional layer;

[0116] Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

[0117] As an optional embodiment, the size of the initial face image is 128×128;

[0118] The size of the first image block is 128×10, the size of the second image block is 128×64, and the size of the third image block is 128×54.

[0119] As an optional embodiment, the size of the first feature map is 256×128, and the number of output channels is 128.

[0120] As an optional embodiment, the size of the maximum pooling operation is 2×2, and the step size is 2; the size of the third feature map is 128×64, and the number of output channels is 128;

[0121] The size of the average pooling operation is 3×3, the stride is 1, and the padding is 1; the size of the second feature map is 128×64, and the number of output channels is 128.

[0122] As an optional embodiment, the convolution kernel size of the first convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fourth feature map is 128×64, and the number of output channels is 64;

[0123] The convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the feature map output by the second convolution layer is 128×64, and the number of output channels is 128; the convolution kernel size of the third convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fifth feature map is 128×64, and the number of output channels is 64.

[0124] As an optional embodiment, the initial face image is a face image detected in face recognition detection, and the apparatus further includes:

[0125] The recognition module is used to perform face matching and recognition using the stitched complete face image without glasses.

[0126] In this embodiment of the present invention, first, upsampling is performed to obtain a higher-resolution feature map, improving the quality of the reconstructed image. Downsampling is then performed, and the image is reconstructed through operations such as convolution, pooling, and merging. Finally, convolution is used to map the feature map back to the original pixel space. This results in fewer network layers, thus maintaining a low computational load and faster forward processing efficiency. Furthermore, a trained convolutional neural network model is used to process facial images, outputting a reconstructed image of the face without glasses, thereby improving facial recognition rates.

[0127] It should be noted that the glasses removal device provided in the embodiment of the present invention is a device capable of executing the above-mentioned glasses removal method. Therefore, all embodiments of the above-mentioned glasses removal method are applicable to the device and can achieve the same or similar beneficial effects.

[0128] like Figure 4 As shown, an embodiment of the present invention further provides a communication device, including a processor 400 and a transceiver 410, wherein the transceiver 410 receives and sends data under the control of the processor 400, and the processor 400 is configured to perform the following operations:

[0129] The initial face image is divided into a first image block, a second image block and a third image block, wherein the second image block includes a complete eyeglass portion;

[0130] Using the second image block as input to a glasses removal shallow convolutional neural network, obtaining a reconstructed glasses-free image block that is the same size as the second image block and output by the glasses removal shallow convolutional neural network; wherein the glasses removal shallow convolutional neural network is used to sequentially perform upsampling and downsampling operations on the second image block, and reconstruct the image through convolution and pooling to obtain the reconstructed glasses-free image block that is the same size as the second image block;

[0131] The reconstructed image block without glasses, the first image block and the third image block are stitched together to obtain a complete face image without glasses.

[0132] As an optional embodiment, the processor is further configured to perform the following process:

[0133] performing an upsampling operation on the second image block using a deconvolution, a bilinear interpolation algorithm, or a super-resolution algorithm to obtain a high-resolution first feature map;

[0134] Reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having the same size as the second image block through a downsampling operation;

[0135] The second feature map is mapped to the original pixel space using a target convolution layer to obtain the reconstructed glasses-free image block.

[0136] As an optional embodiment, the processor is further configured to perform the following process:

[0137] Remove the glasses frames whose brightness is lower than the skin in the first feature map through the maximum pooling operation of the filter, and fill the glasses occluded area with the surrounding skin pixels with higher brightness to obtain a third feature map;

[0138] Performing convolution processing on the third feature map through a dual-branch convolution module to obtain a fourth feature map and a fifth feature map;

[0139] Combining the fourth feature map and the fifth feature map to obtain a sixth feature map;

[0140] The eyeglasses region having a brightness higher than that of the skin in the sixth feature map is removed by an average pooling operation of a filter to obtain the second feature map.

[0141] As an optional embodiment, the processor is further configured to perform the following process:

[0142] Dividing the third feature map into two groups of feature maps along the channel direction;

[0143] Inputting a set of feature maps into the first convolutional layer of the dual-branch convolutional module, and outputting the fourth feature map through the first convolutional layer;

[0144] Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

[0145] As an optional embodiment, the size of the initial face image is 128×128;

[0146] The size of the first image block is 128×10, the size of the second image block is 128×64, and the size of the third image block is 128×54.

[0147] As an optional embodiment, the size of the first feature map is 256×128, and the number of output channels is 128.

[0148] As an optional embodiment, the size of the maximum pooling operation is 2×2, and the step size is 2; the size of the third feature map is 128×64, and the number of output channels is 128;

[0149] The size of the average pooling operation is 3×3, the stride is 1, and the padding is 1; the size of the second feature map is 128×64, and the number of output channels is 128.

[0150] As an optional embodiment, the convolution kernel size of the first convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fourth feature map is 128×64, and the number of output channels is 64;

[0151] The convolution kernel size of the second convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the feature map output by the second convolution layer is 128×64, and the number of output channels is 128; the convolution kernel size of the third convolution layer is 3×3, the stride is 1, and the padding is 1; the size of the fifth feature map is 128×64, and the number of output channels is 64.

[0152] As an optional embodiment, the initial face image is a face image detected in face recognition detection, and the processor is further configured to perform the following process:

[0153] The complete face image without glasses obtained by stitching is used for face matching and recognition.

[0154] In this embodiment of the present invention, first, upsampling is performed to obtain a higher-resolution feature map, improving the quality of the reconstructed image. Downsampling is then performed, and the image is reconstructed through operations such as convolution, pooling, and merging. Finally, convolution is used to map the feature map back to the original pixel space. This results in fewer network layers, thus maintaining a low computational load and faster forward processing efficiency. Furthermore, a trained convolutional neural network model is used to process facial images, outputting a reconstructed image of the face without glasses, thereby improving facial recognition rates.

[0155] It should be noted that the communication device provided in the embodiment of the present invention is a communication device capable of executing the above-mentioned glasses removal method, so all embodiments of the above-mentioned glasses removal method are applicable to the communication device and can achieve the same or similar beneficial effects.

[0156] An embodiment of the present invention further provides a communication device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the various processes in the embodiment of the glasses removal method described above are implemented, and the same technical effects can be achieved. To avoid repetition, these processes will not be described here.

[0157] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When executed by a processor, the program implements the various processes of the aforementioned eyeglass removal method embodiment and achieves the same technical effects. To avoid repetition, the details are omitted here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0158] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0159] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 means for performing functions specified in one or more processes and / or one or more blocks.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause the computer or other programmable device to execute a series of operating steps to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0162] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for removing glasses, characterized in that: include: The initial face image is divided into a first image block, a second image block and a third image block, wherein the second image block includes a complete glasses portion; The second image block is used as an input of a glasses removal shallow convolutional neural network, and a reconstructed glasses-free image block having the same size as the second image block and output by the glasses removal shallow convolutional neural network is obtained; wherein the glasses removal shallow convolutional neural network is used to sequentially perform upsampling operations and downsampling operations on the second image block, and reconstruct the image through convolution and pooling, so as to obtain the reconstructed glasses-free image block having the same size as the second image block; Combine the reconstructed image block without glasses, the first image block and the third image block to obtain a complete face image without glasses; The glasses remove the processing step of the shallow convolutional neural network on the second image block, including: Performing an upsampling operation on the second image block by using deconvolution, a bilinear difference algorithm or a super-resolution algorithm to obtain a high-resolution first feature map; Reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having a size consistent with that of the second image block through a downsampling operation; The second feature map is mapped to the original pixel space using a target convolutional layer to obtain the reconstructed glasses-free image block.

2. The method according to claim 1, characterized in that The method of reconstructing the first feature map by convolution and pooling, and obtaining a second feature map having a size consistent with that of the second image block by downsampling operation, comprises: The glasses frames whose brightness is lower than the skin in the first feature map are removed by the maximum pooling operation of the filter, and the glasses occlusion area is filled with surrounding skin pixels with higher brightness to obtain a third feature map; Performing convolution processing on the third feature map respectively through a double-branch convolution module to obtain a fourth feature map and a fifth feature map; Combining the fourth feature map and the fifth feature map to obtain a sixth feature map; The eyeglasses region having a brightness higher than that of the skin in the sixth feature map is removed by an average pooling operation of a filter to obtain the second feature map.

3. The method according to claim 2, characterized in that The method of performing convolution processing on the third feature map respectively by using a dual-branch convolution module to obtain a fourth feature map and a fifth feature map includes: Dividing the third feature map into two groups of feature maps along the channel direction; Inputting a set of feature maps into the first convolution layer of the dual-branch convolution module, and outputting the fourth feature map through the first convolution layer; Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

4. The method according to claim 3, characterized in that The size of the initial face image is 128×128; The size of the first image block is 128×10, the size of the second image block is 128×64, and the size of the third image block is 128×54.

5. The method according to claim 4, characterized in that The size of the first feature map is 256×128, and the number of output channels is 128.

6. The method according to claim 5, characterized in that The size of the maximum pooling operation is 2×2, and the step size is 2; the size of the third feature map is 128×64, and the number of output channels is 128; The size of the average pooling operation is 3×3, the step size is 1, and the padding is 1; the size of the second feature map is 128×64, and the number of output channels is 128.

7. The method according to claim 6, characterized in that The convolution kernel size of the first convolution layer is 3×3, the step size is 1, and the padding is 1; the size of the fourth feature map is 128×64, and the number of output channels is 64; The convolution kernel size of the second convolution layer is 3×3, the step size is 1, and the padding is 1; the size of the feature map output by the second convolution layer is 128×64, and the number of output channels is 128; the convolution kernel size of the third convolution layer is 3×3, the step size is 1, and the padding is 1; the size of the fifth feature map is 128×64, and the number of output channels is 64.

8. The method according to any one of claims 1 to 7, characterized in that: The initial face image is a face image detected in face recognition detection, and the method further includes: The complete face image without glasses obtained by stitching is used for face matching and recognition.

9. A glasses removal device, characterized in that: include: A segmentation module, used for segmenting the initial face image into a first image block, a second image block and a third image block, wherein the second image block includes a complete glasses portion; a removal module, configured to use the second image block as an input of a glasses removal shallow convolutional neural network, and obtain a reconstructed glasses-free image block output by the glasses removal shallow convolutional neural network and having the same size as the second image block; wherein the glasses removal shallow convolutional neural network is configured to sequentially perform upsampling operations and downsampling operations on the second image block, and reconstruct the image through convolution and pooling, so as to obtain the reconstructed glasses-free image block having the same size as the second image block; A stitching module, used for stitching the reconstructed image block without glasses, the first image block and the third image block to obtain a complete face image without glasses; Wherein, the removal module comprises: A first submodule, configured to perform an upsampling operation on the second image block by using a deconvolution, a bilinear difference algorithm or a super-resolution algorithm to obtain a high-resolution first feature map; A second submodule is used to reconstruct the first feature map through convolution and pooling, and obtain a second feature map having a size consistent with that of the second image block through a downsampling operation; The third submodule is used to use the target convolution layer to map the second feature map to the original pixel space to obtain the reconstructed glasses-free image block.

10. A communication device, comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, characterized in that: The processor is configured to perform the following operations: The initial face image is divided into a first image block, a second image block and a third image block, wherein the second image block includes a complete glasses portion; The second image block is used as an input of a glasses removal shallow convolutional neural network, and a reconstructed glasses-free image block having the same size as the second image block and output by the glasses removal shallow convolutional neural network is obtained; wherein the glasses removal shallow convolutional neural network is used to sequentially perform upsampling operations and downsampling operations on the second image block, and reconstruct the image through convolution and pooling, so as to obtain the reconstructed glasses-free image block having the same size as the second image block; Combine the reconstructed image block without glasses, the first image block and the third image block to obtain a complete face image without glasses; The processor is further configured to execute the following process: Performing an upsampling operation on the second image block by using deconvolution, a bilinear difference algorithm or a super-resolution algorithm to obtain a high-resolution first feature map; Reconstructing the first feature map through convolution and pooling, and obtaining a second feature map having a size consistent with that of the second image block through a downsampling operation; The second feature map is mapped to the original pixel space using a target convolutional layer to obtain the reconstructed glasses-free image block.

11. The communication device according to claim 10, characterized in that The processor is also used to execute the following process: The glasses frames whose brightness is lower than the skin in the first feature map are removed by the maximum pooling operation of the filter, and the glasses occlusion area is filled with surrounding skin pixels with higher brightness to obtain a third feature map; Performing convolution processing on the third feature map respectively through a double-branch convolution module to obtain a fourth feature map and a fifth feature map; Combining the fourth feature map and the fifth feature map to obtain a sixth feature map; The eyeglasses region having a brightness higher than that of the skin in the sixth feature map is removed by an average pooling operation of a filter to obtain the second feature map.

12. The communication device according to claim 11, characterized in that The processor is also used to execute the following process: Dividing the third feature map into two groups of feature maps along the channel direction; Inputting a set of feature maps into the first convolution layer of the dual-branch convolution module, and outputting the fourth feature map through the first convolution layer; Another set of feature maps is input into the second convolution layer and the third convolution layer of the dual-branch convolution module, and the fifth feature map is output through the third convolution layer.

13. A communication device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the glasses removal method according to any one of claims 1 to 8 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the glasses removal method according to any one of claims 1 to 8 are implemented.

Images