A near infrared face image generation method

Abstract

The present invention discloses a near-infrared face image generation method, and relates to the field of image processing technology. The method includes: collecting a face image and graying it; constructing a cyclic generative adversarial network; inputting the face image into the constructed cyclic generative adversarial network for training to obtain a final cyclic generative adversarial network; inputting the grayscale image of the face to be processed into the final cyclic generative adversarial network to generate a near-infrared face image. The present invention converts the face RGB image into a face grayscale image and then inputs it into the cyclic generative adversarial network, thereby avoiding the noise caused by the color spectrum information in the RGB image and making the network learning direction correct; and the present invention adds a residual module between downsampling and upsampling, and merges the feature map output by the downsampling layer with the restoration map output by the upsampling layer through a jump connection, so that details and structures can be better restored during the upsampling process, thereby improving the visual quality of the generated near-infrared face image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for generating a near-infrared face image. Background Art

[0002] At present, the widespread application of face detection and liveness detection algorithms has become an important application in the field of science and technology. The rise of these algorithms is also accompanied by an urgent demand for face datasets and near-infrared face datasets. Near-infrared technology has unique advantages in overcoming problems such as illumination changes and occlusion. The establishment of such datasets is crucial to improving the performance of algorithms in different environments. Compared with traditional visible light face datasets, the types of near-infrared face datasets currently available for public use are relatively limited. This is due to the special nature of its acquisition technology and the high difficulty of shooting. This limits researchers and developers from studying and improving near-infrared face recognition algorithms. Therefore, the establishment of more abundant near-infrared face datasets is crucial to promoting the development of related fields.

[0003] Most current research and patents attempt to obtain near-infrared face images corresponding to visible light faces by designing special hardware cameras. Although these methods are innovative in trying to solve the problem of insufficient near-infrared face data, they still have some limitations and challenges. On the one hand, the amount of data obtained by these methods is limited and may not meet large-scale data needs. On the other hand, obtaining near-infrared face images by designing special hardware cameras may require high costs, and may face certain economic pressures from design, production to deployment. This may limit the feasibility and popularity of these methods in practical applications.

[0004] In addition, there are a small number of studies or patents that use algorithms to convert visible light images and near-infrared images. The goal of these algorithms is to establish a connection between two different spectral ranges so that images can be converted between visible light and near-infrared. Some of these methods use convolutional neural networks to achieve image conversion, but there are serious problems with the generalization ability of such methods; specifically, these methods require pairing near-infrared images with visible light images to establish training data for mutual conversion. Such paired data may be difficult to obtain in practical applications because there may be differences between images under different spectral conditions, including differences in lighting, texture, and shape. Due to the difficulty in data pairing, the trained model may not be able to generalize well in the real world, resulting in poor performance in unseen situations. There are also some methods that use adversarial generative networks (GANs) for spectral image conversion problems. These methods aim to achieve style conversion by learning the imaging characteristics of two different types of images, thereby converting images from one spectral range to another. Unlike the previously mentioned methods, adversarial generative networks (GANs) do not require pre-paired data and also have certain generalization capabilities. However, due to the huge difference in style between visible light face images and near-infrared face images, these GAN networks need to redesign the optimization direction and adjust the weights of the loss components of the model when doing this application.

[0005] In the invention with patent number CN112347850A, a cyclic adversarial generative network CycleGAN is used to generate near-infrared images. The network consists of two parts: a generator and a discriminator. The generator attempts to generate realistic near-infrared images, while the discriminator strives to distinguish the generated images from real near-infrared images. and It is used to match two types of images and bring the pixels of the two types of images closer together through the regression loss function. In addition, in order to ensure that the image is not distorted, a discriminator is used. and To reshape the correlation between pixels, such as Figure 1 shown.

[0006] However, there are some problems in directly using the CycleGAN to perform style transfer on visible light images and near-infrared images: first, visible light faces have color information and are three-channel images, while near-infrared face images focus more on spectral information and are closer to single-channel grayscale images. The output image obtained by inputting the three-channel visible light face image into the CycleGAN is also three-channel, and a single-channel grayscale image cannot be obtained. In addition, the generator in the CycleGAN uses a convolutional layer for downsampling to capture the global features of the image, and then uses a deconvolutional layer for upsampling to restore details. However, in this process, downsampling may result in loss of details and local information, resulting in inaccurate recovery of the upsampling, resulting in low visual quality of the infrared face image.

[0007] In summary, there are the following problems when using the cyclic adversarial generative network CycleGAN to generate near-infrared face images: it is impossible to generate a single-channel near-infrared face image using a color visible light face image, and downsampling may result in loss of details and local information, resulting in inaccurate recovery of upsampling. Summary of the invention

[0008] Based on this, it is necessary to provide a near-infrared face image generation method to address the above technical problems.

[0009] This manual adopts the following technical solutions:

[0010] This specification provides a near infrared face image generation method, including:

[0011] Collect near-infrared face images and visible light face RGB images, and convert the visible light face RGB images into visible light face grayscale images;

[0012] A cyclic generative adversarial network for generating near-infrared face images is constructed, wherein the cyclic generative adversarial network includes a first generator Gx, a second generator Gy, a first discriminator Dx, and a second discriminator Dy, wherein the first discriminator Dx is connected to the output end of the first generator Gx, and the second discriminator Dy is connected to the output end of the second generator Gy; the first generator Gx and the second generator Gy have the same structure, and both include a downsampling module, a residual block module, and an upsampling module;

[0013] The downsampling module is used to downsample the input image through several downsampling layers to obtain feature maps of different sizes; the residual block module is used to extract features from the feature map of the minimum size through several residual blocks; the upsampling module is used to upsample the output of the residual block module through several upsampling layers to obtain recovery maps of different sizes; the feature map output by each downsampling layer is merged with the recovery map output by the corresponding upsampling layer through a jump connection as the input of the next upsampling layer;

[0014] Input the near-infrared face image and the visible light face grayscale image into the constructed cyclic generative adversarial network for training to obtain the final cyclic generative adversarial network;

[0015] The grayscale image of the face to be processed is input into the final cyclic generative adversarial network to generate a near-infrared face image.

[0016] Furthermore, it also includes:

[0017] Use bilinear interpolation to convert the low-resolution visible light face RGB image into a high-resolution visible light face RGB image:

[0018] (1)

[0019] (2)

[0020] (3)

[0021] (4)

[0022] (5)

[0023] in,

[0024] I'(x', y') represents the pixel value at coordinate (x', y') on the generated high-resolution visible light face RGB image, and I(x, y) represents the pixel value at coordinate (x, y) on the low-resolution visible light face RGB image;

[0025] (x1, y1), (x1, y2), (x2, y1), (x2, y2) are four points on the low-resolution visible light face RGB image;

[0026] Formula (5) means that the pixel value of each point on the high-resolution visible light face RGB image is interpolated using the four points on the low-resolution visible light face RGB image in a weighted average manner.

[0027] Furthermore, converting the visible light human face RGB image into a visible light human face grayscale image includes:

[0028] (6)

[0029] in,

[0030] g(x', y') represents the gray value at coordinate (x', y') on the visible light face gray image;

[0031] R(x', y'), G(x', y'), and B(x', y') represent the pixel values ​​at the coordinates (x', y') of the red, green, and blue channels in the visible light face RGB image, respectively.

[0032] Furthermore, the feature maps of different sizes include:

[0033] (7)

[0034] (8)

[0035] in,

[0036] , are the height and width of the output feature map of the i-th downsampling layer;

[0037] , They are the height and width of the output feature map of the i-1th downsampling layer, respectively;

[0038] is the size of the downsampling window of the i-th downsampling layer;

[0039] is the downsampling step size of the i downsampling layers.

[0040] Furthermore, the upsampling layer comprises:

[0041] A pixel shuffle layer for upsampling by rearranging the pixels in the restored image output by the previous upsampling layer; a convolutional layer for feature extraction;

[0042] The pixel shuffling layer comprises:

[0043] Determine the upsampling ratio of the jth upsampling layer ;

[0044] According to the upsampling factor, create a new blank image with the size:

[0045] (9)

[0046] (10)

[0047] in,

[0048] , are the height and width of the output recovery image of the jth upsampling layer, respectively;

[0049] , They are the height and width of the output recovery image of the j-1th upsampling layer, respectively;

[0050] The restored image output by the j-1th upsampling layer is evenly divided into ( ) image blocks;

[0051] For each image block, a bilinear interpolation operation is performed to generate new pixel values;

[0052] The new pixel values ​​are filled into the corresponding positions in the created new blank image to obtain the pixel shuffling result.

[0053] Furthermore, the near-infrared face image and the visible light face grayscale image are input into the constructed cyclic generative adversarial network for training, including:

[0054] Preprocess the near-infrared face images and visible light face grayscale images by cropping, standardizing, and enhancing, and make the preprocessed visible light face grayscale images into domain X datasets, and make the preprocessed near-infrared face images into domain Y datasets, and randomly select a batch of images from domain X datasets and domain Y datasets as training samples;

[0055] Initialize the weight parameters of the cyclic generative adversarial network, input the domain X training image into the first generator Gx, and generate the corresponding domain Y fake image fake_Y according to the initialized weight parameters; input the domain Y training image into the second generator Gy, and generate the corresponding domain X fake image fake_X according to the initialized weight parameters;

[0056] The generated fake image fake_X is input into the first discriminator Dx, and the first discriminator Dx discriminates whether the input image is a real domain X training image or a generated fake image fake_X, and calculates the first adversarial loss function according to the discrimination result; the generated fake image fake_Y is input into the second discriminator Dy, and the second discriminator Dy discriminates whether the input image is a real domain Y training image or a generated fake image fake_Y, and calculates the second adversarial loss function according to the discrimination result;

[0057] The generated fake image fake_X is input into the second generator Gy to obtain the recall image recall_X of the domain X training image, and the first cycle consistency loss function is calculated according to the gap between the domain X training image and the corresponding recall image recall_X; the generated fake image fake_Y is input into the first generator Gx to obtain the recall image recall_Y of the domain Y training image, and the second cycle consistency loss function is calculated according to the gap between the domain Y training image and the corresponding recall image recall_Y;

[0058] The total loss function is obtained according to the first adversarial loss function, the second adversarial loss function, the first cycle consistency loss function, and the second cycle consistency loss function. The initialized weight parameters are iteratively updated through the stochastic gradient descent optimization algorithm to obtain the weight parameters that minimize the total loss function, and the trained cyclic generative adversarial network is obtained.

[0059] Further, the obtaining of the total loss function according to the first adversarial loss function, the second adversarial loss function, the first cycle consistency loss function, and the second cycle consistency loss function includes:

[0060] (11)

[0061] (12)

[0062] (13)

[0063] (14)

[0064] in,

[0065] is the total loss function, is the first adversarial loss function, is the second adversarial loss function, is the weight coefficient, is the total consistency loss function, is the first cycle consistency loss function, Second cycle consistency loss function;

[0066] X represents the domain X training image, which is the preprocessed visible light face grayscale image; Y represents the domain Y training image, which is the preprocessed near infrared face image;

[0067] It is the fake image fake_Y generated by the first generator Gx; is the discrimination result of the second discriminator Dy on whether the input real domain Y training image is a real domain Y training image or a generated fake image fake_Y; is the discrimination result of the second discriminator Dy on whether the input real domain Y training image is a real domain Y training image or a generated fake image fake_Y;

[0068] It is the fake image fake_X generated by the second generator Gy; It is the discrimination result of the first discriminator Dx on whether the input real domain X training image is a real domain X training image or a generated fake image fake_X; It is the discrimination result of the first discriminator Dx on whether the input real fake image fake_X is a real domain X training image or a generated fake image fake_X.

[0069] Furthermore, the step of inputting the grayscale face image to be processed into the final cyclic generative adversarial network to generate a near-infrared face image includes:

[0070] The grayscale face image to be processed is input into the first generator Gx and converted into a near-infrared face image.

[0071] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0072] Based on the characteristic that near-infrared face images are near-grayscale images, the present invention converts visible light face RGB images into visible light face grayscale images and then inputs them into a recurrent generative adversarial network, so that the recurrent generative adversarial network can directly learn the contour and grayscale spectrum of the near-grayscale image from the visible light face grayscale image, avoiding the noise caused by the color spectrum information in the RGB image, and making the recurrent generative adversarial network learn in the correct direction.

[0073] In addition, in the generator of the recurrent generative adversarial network constructed by the present invention, a residual module is added between the downsampling module and the upsampling module. The residual module extracts features of the feature map of the minimum size through a plurality of residual blocks. The residual connection mechanism in the residual block can effectively capture the subtle differences and detail information in the feature map by learning the residual changes between the input and output. The feature map obtained by the residual module contains the detail information of the key input image. Then, the feature map obtained by the residual module is upsampled. During the upsampling process, the feature map output by each downsampling layer is merged with the restoration map output by the corresponding upsampling layer through a jump connection as the input of the next upsampling layer, which helps to retain the details and spatial information in the feature map, so that the details and structures can be better restored during the upsampling process, thereby improving the visual quality of the generated near-infrared face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0075] Figure 1 The method schematic diagram provided for this specification is patent number: CN112347850A, Figure 1 The left is the original face grayscale image. Figure 1 Right is based on Figure 1 Left: the generated near-infrared face image. Figure 1 The left one comes from the public dataset The casianir-vis 2.0 face database;

[0076] Figure 2 A flow chart of a near-infrared face image generation method provided in this specification;

[0077] Figure 3 A schematic diagram of a single linear interpolation provided in this specification;

[0078] Figure 4 A cubic single linear interpolation schematic diagram provided in this specification;

[0079] Figure 5 A traditional visible light face RGB processing flow chart provided in this manual;

[0080] Figure 6 A traditional near-infrared face image processing flow chart provided in this manual;

[0081] Figure 7 A visible light face grayscale image processing flow chart of the present invention provided in this specification;

[0082] Figure 8 A near infrared face image processing flow chart of the present invention provided in this specification;

[0083] Fig. 9 A schematic diagram of a jump connection provided in this specification;

[0084] Fig.10 This is a schematic diagram of the near-infrared face image generation result of a traditional solution provided in this manual. Fig.10 (a) is the original face grayscale image, Fig.10 (b) is based on Fig.10 (a) Generated near-infrared face image, Fig.10 (a) From the public dataset The casia nir-vis 2.0 face database;

[0085] Fig.11 This is a schematic diagram of the near-infrared face image generation result of a traditional solution provided in this manual. Fig.11 (a) is the original face grayscale image, Fig.11 (b) is based on Fig.11 (a) Generated near-infrared face image, Fig.11 (a) From the public dataset The casia nir-vis 2.0 face database;

[0086] Fig.12 This is a schematic diagram of the near-infrared face image generation result of the present invention provided in this specification. Fig.12 (a) is the original face grayscale image, Fig.12 (b) is based on Fig.12 (a) Generated near-infrared face image, Fig.12 (a) From the public dataset The casia nir-vis 2.0 face database;

[0087] Fig.13 This is a schematic diagram of the near-infrared face image generation result of the present invention provided in this specification. Fig.13 (a) is the original face grayscale image, Fig.13 (b) is based on Fig.13 (a) Generated near-infrared face image, Fig.13 (a) From the public dataset The casia nir-vis 2.0 face database;

[0088] Fig.14 A schematic diagram of a one-dimensional chessboard artifact provided for this specification;

[0089] Fig.15 A schematic diagram of a two-dimensional chessboard artifact provided for this specification;

[0090] Fig.16 This is a schematic diagram of the near-infrared face image generation result of the present invention provided in this specification. Fig.16 (a) Fig.16 (b) is the generated near-infrared face image with chessboard artifacts. Fig.16 (c) is the generated near-infrared face image without chessboard artifacts. Fig.16 (a) Fig.16 (b) Fig.16 (c)Derived from the public dataset The casia nir-vis 2.0 facedatabase.

[0091] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0092] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0093] Figure 2 This is a flowchart of a near infrared face image generation method in this specification, which specifically includes the following steps:

[0094] Step 1: Generate high-resolution visible light images

[0095] The present invention uses bilinear interpolation to generate high-resolution visible light images.

[0096] First, we introduce single linear interpolation, such as Figure 3 As shown in the figure, when the coordinates of the middle point P between P1 and P2 are required, the following formula is used:

[0097] (1)

[0098] Accordingly, when the need for Q 11 (x1, y1), Q 12 (x1, y2), Q 21 (x2, y1), Q 22 When calculating the coordinates of a point P between (x2, y2), we can first use two single linear interpolations to obtain the coordinates of R1 and R2 respectively, such as Figure 4 Then, use single linear interpolation on R1 and R2 to obtain the coordinates of P. The cubic single linear interpolation formula is as follows:

[0099] (2)

[0100] (3)

[0101] (4)

[0102] Bring in

[0103] (5)

[0104] in, , , , , , , , .

[0105] Also because ; ,Will ; Substituting it into formula (5) we can get:

[0106] (6)

[0107] in,

[0108] I'(x', y') represents the pixel value at coordinate (x', y') on the generated high-resolution visible light face RGB image, and I(x, y) represents the pixel value at coordinate (x, y) on the low-resolution visible light face RGB image;

[0109] (x1, y1), (x1, y2), (x2, y1), (x2, y2) are four points on the low-resolution visible light face RGB image; w1 = (x2 - x')(y2 - y'), w2 = (x' - x1)(y2 - y'), w3 = (x2 - x')(y' - y1), w4 = (x' - x1)(y' - y1);

[0110] Formula (6) means that the pixel value of each point on the high-resolution visible light face RGB image is interpolated using the four points on the low-resolution visible light face RGB image by weighted averaging.

[0111] Step 2: Design a recurrent generative adversarial network

[0112] Based on the existing solutions, the main algorithm logic of the present invention still adopts the cyclic adversarial generative network structure, and fits the styles of two different types of images through the cyclic adversarial generative network. However, the traditional cyclic adversarial generative network is more suitable for color image style conversion, and is not suitable for establishing the relationship between visible light faces and near-infrared faces. Problems such as color confusion, face contour distortion, and chessboard artifacts will occur in the generated image. In response to the above problems, the present invention redesigned the algorithm logic and adjusted the backbone network structure.

[0113] like Figure 5 , Figure 6 As shown in the figure, the existing algorithm for near-infrared face image generation using recurrent adversarial networks is based on color RGB face images for network training, but such a design cannot achieve the generation of near-grayscale images such as near-infrared faces. The specific details are analyzed as follows:

[0114] The algorithm logic of the recurrent adversarial generative network is divided into two parts. The first part is to input the visible light face image into the network Get a pseudo infrared face image and then pass it through the network Output visible light face recall image; the other part is similar, also through the generator network and Realize the output recall process of near infrared face. In addition, the discriminator and The function of the algorithm is to distinguish the similarity between the input image and the recall image. The network obtains the output and obtains the generated near-infrared face image. and Keep the same frequency optimization, in the same step and The optimization process is in the same black box, which cannot be guaranteed to be achieved only through The image obtained must be a near-grayscale near-infrared image. In the process of reproducing existing patents and research, the near-infrared image generated by this algorithm has problems such as color spots and facial deformation, such as Fig.10 , Fig.11 Therefore, in the above process, when the goal of the recurrent adversarial generation network is to generate a standard near-infrared face image, the RGB color information can be regarded as a kind of interference or noise and has nothing to do with the generated image.

[0115] Step 2.1: Convert the visible light face RGB image to a visible light face grayscale image

[0116] like Figure 7 , Figure 8 As shown, the present invention completes the entire algorithm logic starting from a single-channel grayscale image. First, in the process of generating pseudo-grayscale from visible light faces, the color visible light image is processed into a three-channel grayscale image. The specific method is to normalize each channel and then connect them, so that some spectral information can be retained. In addition, in the process of generating near-infrared recall face images from near-infrared face images, although the image output of this stage will not be used as the result of the algorithm, the weight update of this stage will still affect the network. and , so the discriminator at this stage The label target also needs to be grayed out to keep the optimization direction consistent. The graying formula is:

[0117] (7)

[0118] in,

[0119] g(x', y') represents the gray value at coordinate (x', y') on the visible light face gray image;

[0120] R(x', y'), G(x', y'), and B(x', y') represent the pixel values ​​at the coordinates (x, y) of the red, green, and blue channels in the visible light face RGB image, respectively.

[0121] Step 2.2: Obtain a Cyclic Generative Adversarial Network

[0122] The generator is the key to the recurrent generative adversarial network. Many similar algorithms for converting visible light faces to near-infrared faces have added some network structures at this stage, such as adding a Transformer module. Face images are relatively simple images. The Transformer module is used to establish the relationship between global pixels. Although it can improve the quality of face generation to a small extent, it will bring exponentially more computational workload to the entire algorithm. Considering that almost all application scenarios of faces have timeliness requirements, this method is difficult to apply in practice. In addition, this type of method cannot solve the problem of chessboard artifacts in image generation.

[0123] Step 2.2.1. Build the generator

[0124] The basic structural process of the generator is: first downsample the input image, obtain the feature vector, and then upsample the image to restore the original size. In the image restoration process, the upsampling strategy uses a deconvolution operation, which will repeatedly restore pixels at fixed positions at different times due to the convolution step size and convolution kernel size, and will bring checkerboard-shaped interference to the generated image. This checkerboard noise will seriously affect the availability of near-infrared face datasets.

[0125] like Fig. 9 As shown, the present invention is a generator and The whole process is improved by adding jump connections to the feature maps generated in the upsampling process to improve the restoration of image generation, so as to improve the quality of generated images while maintaining the algorithm calculation amount; the problem of chessboard artifacts is solved by shuffling pixels. The constructed generator includes downsampling module, residual block module, and upsampling module. The specific details are as follows:

[0126] First, the downsampling module downsamples the input image through several downsampling layers to obtain feature maps of different sizes. This feature map extracts the correlation information between pixels in the face image. The current feature map is saved each time the downsampling is performed. Compared with the feature map at the bottom layer, the feature map at the front of the network contains more complete image details. These details can be used to restore the image later to add contour information to the face. The sizes of each feature map obtained are:

[0127] (8)

[0128] (9)

[0129] in,

[0130] , are the height and width of the output feature map of the i-th downsampling layer;

[0131] , They are the height and width of the output feature map of the i-1th downsampling layer, respectively;

[0132] is the size of the downsampling window of the ith downsampling layer. The present invention uses a depthwise separable convolution with a convolution kernel of 3 for downsampling, and the downsampling window is 3;

[0133] is the downsampling step size of the i downsampling layers.

[0134] Then the residual block module extracts features from the feature map of the smallest size through 15 residual blocks. These modules do not perform downsampling operations but only extract features. The residual connection mechanism in the residual block can effectively capture the subtle differences and detail information in the feature map by learning the residual changes between the input and output. The feature map obtained by the residual module contains the key detail information of the input image. The feature map after 15 residual blocks has stronger relationship information between the pixels of the original image.

[0135] Then the upsampling module upsamples the output of the residual block module through several upsampling layers to obtain recovery images of different sizes. The upsampling layer of the present invention includes a pixel shuffling layer and a convolution layer. Pixel shuffling can be regarded as a special shape transformation, which achieves upsampling by moving pixels from the channel dimension to the length and width dimension. In this way, each pixel of the upsampled image comes from the previous feature map, which can effectively avoid the stacking problem of deconvolution. The pixel shuffling layer includes:

[0136] Determine the upsampling ratio of the jth upsampling layer ;

[0137] According to the upsampling factor, create a new blank image with the size:

[0138] (10)

[0139] (11)

[0140] in,

[0141] , are the height and width of the output recovery image of the jth upsampling layer, respectively;

[0142] , They are the height and width of the output recovery image of the j-1th upsampling layer, respectively;

[0143] The restored image output by the j-1th upsampling layer is evenly divided into ( ) image blocks;

[0144] For each image block, a bilinear interpolation operation is performed to generate new pixel values;

[0145] The new pixel values ​​are filled into the corresponding positions in the created new blank image to obtain the pixel shuffling result.

[0146] In the upsampling process, the feature map output by each downsampling layer is merged with the recovery map output by the corresponding upsampling layer through a skip connection. The scale of the feature map and the recovery map must be consistent during the skip connection. However, the last layer of the network does not use a skip residual connection because it will bring in a lot of information related to the original image, which will affect the algorithm to generate the image.

[0147] Step 2.2.2. Design loss function

[0148] (12)

[0149] (13)

[0150] (14)

[0151] (15)

[0152] in,

[0153] is the total loss function, is the first adversarial loss function, is the second adversarial loss function, is the weight coefficient, is the total consistency loss function, is the first cycle consistency loss function, Second cycle consistency loss function;

[0154] X represents the real domain X image, which is the preprocessed visible light face grayscale image; Y represents the real domain Y image, which is the preprocessed near-infrared face image;

[0155] It is the fake image fake_Y generated by the first generator Gx; is the discrimination result of the second discriminator Dy on whether the input real domain Y training image is a real domain Y training image or a generated fake image fake_Y; is the discrimination result of the second discriminator Dy on whether the input real domain Y training image is a real domain Y training image or a generated fake image fake_Y;

[0156] It is the fake image fake_X generated by the second generator Gy; It is the discrimination result of the first discriminator Dx on whether the input real domain X training image is a real domain X training image or a generated fake image fake_X; It is the discrimination result of the first discriminator Dx on whether the input real fake image fake_X is a real domain X training image or a generated fake image fake_X.

[0157] Step 2.2.3: Training the Generative Adversarial Network, including:

[0158] Preprocess the near-infrared face images and visible light face grayscale images by cropping, standardizing, and enhancing, and make the preprocessed visible light face grayscale images into domain X datasets, and make the preprocessed near-infrared face images into domain Y datasets, and randomly select a batch of images from domain X datasets and domain Y datasets as training samples;

[0159] Initialize the weight parameters of the cyclic generative adversarial network, input the domain X training image into the first generator Gx, and generate the corresponding domain Y fake image fake_Y according to the initialized weight parameters; input the domain Y training image into the second generator Gy, and generate the corresponding domain X fake image fake_X according to the initialized weight parameters;

[0160] The generated fake image fake_X is input into the first discriminator Dx, and the first discriminator Dx discriminates whether the input image is a real domain X training image or a generated fake image fake_X, and calculates the first adversarial loss function according to the discrimination result; the generated fake image fake_Y is input into the second discriminator Dy, and the second discriminator Dy discriminates whether the input image is a real domain Y training image or a generated fake image fake_Y, and calculates the second adversarial loss function according to the discrimination result;

[0161] The generated fake image fake_X is input into the second generator Gy to obtain the recall image recall_X of the domain X training image, and the first cycle consistency loss function is calculated according to the gap between the domain X training image and the corresponding recall image recall_X; the generated fake image fake_Y is input into the first generator Gx to obtain the recall image recall_Y of the domain Y training image, and the second cycle consistency loss function is calculated according to the gap between the domain Y training image and the corresponding recall image recall_Y;

[0162] The total loss function is obtained according to the first adversarial loss function, the second adversarial loss function, the first cycle consistency loss function, and the second cycle consistency loss function. The initialized weight parameters are iteratively updated through the stochastic gradient descent optimization algorithm to obtain the weight parameters that minimize the total loss function, and the trained cyclic generative adversarial network is obtained.

[0163] Step 3: Input the grayscale face image to be processed into the first generator Gx and convert it into a near-infrared face image.

[0164] In addition, this specification also provides comparative examples of this solution and other solutions.

[0165] 1. Training effect using grayscale visible light face images

[0166] To achieve style conversion between visible light face images and near-grayscale near-infrared face images, we should focus on the face contour and the grayscale-based spectral information in the near-infrared face image, while the color RGB information in the visible light face image will become noise and affect the generator. and and discriminator and Parameter optimization. Therefore, in order to avoid this RGB interference and make the generator focus on the face contour and grayscale spectrum information, the present invention designs an algorithm for generating near-grayscale near-infrared face generators from grayscale visible light faces. Compared with some existing algorithms, the near-infrared face images generated by the present invention have a very high degree of restoration, and some facial features with infrared characteristics are restored very well. The following will be described in detail.

[0167] Firstly, based on the design ideas of the existing technology, the algorithms for converting most of the current visible light face images into near-infrared face images were restored. These algorithms start from RGB color visible light images and implement the training process. Fig.10 As shown, color spots are generated during the generation of near-infrared face images; Fig.11 In the above, the imaging characteristics of the generated near-infrared face image are opposite to those of the actual near-infrared face image, and the cheekbones of the face should be the bright part.

[0168] The generation algorithm was redesigned based on the grayscale visible light face, which makes the algorithm pay more attention to the contours of the face and the grayscale spectrum information. Figure 12-13 In the example above, the generator algorithm responds very well to grayscale images. In addition, starting from grayscale visible light faces, we can learn and generate more detailed near-infrared images.

[0169] 2. Solve the checkerboard artifacts by pixel shuffling

[0170] When using neural networks for generative tasks, there is usually a step to build an image from a low resolution, known as upsampling. This allows the network to describe a rough image and then fill in the details. In order to do this, some method is needed to convert the lower resolution image to a higher resolution image, which is usually done through deconvolution. Roughly speaking, the deconvolution layer uses each point in the small image to "draw" a square in the large image, and when the convolution kernel size cannot be divided by the stride, the deconvolution will have uneven overlap. This phenomenon will be more serious in two dimensions, and the stacking will occur both horizontally and vertically, such as Fig.14 , Fig.15 As shown in , this is reflected in the generated image as layers of chessboard-like spots, such as Fig.16 as shown in .

[0171] To solve this problem, the most intuitive solution is to replace deconvolution with other upsampling methods. This patent uses pixel shuffling plus convolution layer to replace deconvolution. Pixel shuffling can be regarded as a special shape transformation, which achieves upsampling by moving pixels from the channel dimension to the length and width dimension. In this way, each pixel of the upsampled image comes from the previous feature map, which can effectively avoid the stacking problem of deconvolution.

[0172] 3. Model calculation amount and inference speed

[0173] The existing visible light to face image to near-infrared face image algorithms based on recurrent adversarial neural networks have been improved to varying degrees, but many studies or inventions have added Transformer modules. Although this module has achieved good results in the field of computer vision, its computational complexity makes it difficult to apply to real-time scenarios, especially tasks such as face detection. The present invention proposes a recurrent adversarial generative network model based on an improved residual module, which can achieve a similar effect without using the Transformer module, while the computational complexity and parameter amount of the model are less than one-eighth of the recurrent adversarial neural network of the Transformer.

[0174] Record the amount of computation consumed to generate a near-infrared face image from a visible light face image, that is, the input visible light face is passed through the network At the same time, ablation experiments were conducted to compare the network parameters, as shown in Table 1:

[0175] Table 1 Comparison of model calculation amount

[0176]

[0177] The image size of each model input is 128x128, and the inference test is performed on the same CPU, which is Intel(R) Core(TM) i5-10400F.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A near-infrared face image generation method, characterized in that: include: Collect near-infrared face images and visible light face RGB images, and convert the visible light face RGB images into visible light face grayscale images; Use bilinear interpolation to convert a low-resolution visible light face RGB image into a high-resolution visible light face RGB image: w1=(x2-x')(y2-y') (1) w2=(x'-x1)(y2-y') (2) w3=(x2-x')(y'-y1) (3) w4=(x'-x1)(y'-y1) (4) I'(x',y')=w1*I(x1,y1)+w2*I(x1,y2)+w3*I(x2,y1)+w4*I(x2,y2) (5) in, I'(x',y') represents the pixel value at the coordinate (x',y') on the generated high-resolution visible light face RGB image, and I(x,y) represents the pixel value at the coordinate (x,y) on the low-resolution visible light face RGB image; (x1,y1), (x1,y2), (x2,y1), (x2,y2) are four points on the low-resolution visible light face RGB image; The meaning of formula (5) is to interpolate the pixel value of each point on the high-resolution visible light face RGB image using the four points on the low-resolution visible light face RGB image in a weighted average manner; A cyclic generative adversarial network for generating near-infrared face images is constructed, wherein the cyclic generative adversarial network includes a first generator Gx, a second generator Gy, a first discriminator Dx, and a second discriminator Dy, wherein the first discriminator Dx is connected to the output end of the first generator Gx, and the second discriminator Dy is connected to the output end of the second generator Gy; the first generator Gx and the second generator Gy have the same structure, and both include a downsampling module, a residual block module, and an upsampling module; The downsampling module is used to downsample the input image through several downsampling layers to obtain feature maps of different sizes; the residual block module is used to extract features from the feature map of the minimum size through several residual blocks; the upsampling module is used to upsample the output of the residual block module through several upsampling layers to obtain recovery maps of different sizes; the feature map output by each downsampling layer is merged with the recovery map output by the corresponding upsampling layer through a jump connection as the input of the next upsampling layer; The upsampling layer comprises: A pixel shuffle layer for upsampling by rearranging the pixels in the restored image output by the previous upsampling layer; a convolutional layer for feature extraction; The pixel shuffling layer comprises: Determine the upsampling ratio u of the jth upsampling layer j ; According to the upsampling factor, create a new blank image with the size: in, are the height and width of the output recovery image of the jth upsampling layer, respectively; They are the height and width of the output recovery image of the j-1th upsampling layer, respectively; The restored image output by the j-1th upsampling layer is evenly divided into pieces of size (u j ×u j ) image block; For each image block, a bilinear interpolation operation is performed to generate new pixel values; Fill the new pixel values ​​into the corresponding positions in the created new blank image to obtain the pixel shuffling result; Input the near-infrared face image and the visible light face grayscale image into the constructed cyclic generative adversarial network for training to obtain the final cyclic generative adversarial network; The grayscale image of the face to be processed is input into the final cyclic generative adversarial network to generate a near-infrared face image.

2. A near-infrared face image generation method as claimed in claim 1, characterized in that: The step of converting the visible light human face RGB image into a visible light human face grayscale image comprises: g(x',y')=0.2989*R(x',y')+0.5870*G(x',y')+0.1140*B(x',y') (6) in, g(x',y') represents the gray value at coordinate (x',y') on the visible light face gray image; R(x',y'), G(x',y'), and B(x',y') represent the pixel values ​​at the coordinates (x',y') of the red, green, and blue channels in the visible light face RGB image, respectively.

3. A near-infrared face image generation method as claimed in claim 1, characterized in that: The feature maps of different sizes include: in, are the height and width of the output feature map of the i-th downsampling layer; They are the height and width of the output feature map of the i-1th downsampling layer, respectively; is the size of the downsampling window of the i-th downsampling layer; d i is the downsampling step size of the i downsampling layers.

4. A near-infrared face image generation method as claimed in claim 1, characterized in that: The near-infrared face image and the visible light face grayscale image are input into the constructed cyclic generative adversarial network for training, including: Preprocess the near-infrared face images and visible light face grayscale images by cropping, standardizing, and enhancing operations, make the preprocessed visible light face grayscale images into a domain X dataset, and make the preprocessed near-infrared face images into a domain Y dataset, and randomly select a batch of images from the domain X dataset and the domain Y dataset as training samples; Initialize the weight parameters of the cyclic generative adversarial network, input the domain X training image into the first generator Gx, and generate the corresponding domain Y fake image fake_Y according to the initialized weight parameters; input the domain Y training image into the second generator Gy, and generate the corresponding domain X fake image fake_X according to the initialized weight parameters; The generated fake image fake_X is input into the first discriminator Dx, and the first discriminator Dx discriminates whether the input image is a real domain X training image or a generated fake image fake_X, and calculates the first adversarial loss function according to the discrimination result; the generated fake image fake_Y is input into the second discriminator Dy, and the second discriminator Dy discriminates whether the input image is a real domain Y training image or a generated fake image fake_Y, and calculates the second adversarial loss function according to the discrimination result; The generated fake image fake_X is input into the second generator Gy to obtain the recall image recall_X of the domain X training image, and the first cycle consistency loss function is calculated according to the gap between the domain X training image and the corresponding recall image recall_X; the generated fake image fake_Y is input into the first generator Gx to obtain the recall image recall_Y of the domain Y training image, and the second cycle consistency loss function is calculated according to the gap between the domain Y training image and the corresponding recall image recall_Y; The total loss function is obtained according to the first adversarial loss function, the second adversarial loss function, the first cycle consistency loss function, and the second cycle consistency loss function. The initialized weight parameters are iteratively updated through the stochastic gradient descent optimization algorithm to obtain the weight parameters that minimize the total loss function, and the trained cyclic generative adversarial network is obtained.

5. A near-infrared face image generation method as claimed in claim 4, characterized in that: The obtaining of the total loss function according to the first adversarial loss function, the second adversarial loss function, the first cycle consistency loss function, and the second cycle consistency loss function includes: in, is the total loss function, is the first adversarial loss function, is the second adversarial loss function, λ is the weight coefficient, is the total consistency loss function, is the first cycle consistency loss function, Second cycle consistency loss function; X represents the domain X training image, which is the preprocessed visible light face grayscale image; Y represents the domain Y training image, which is the preprocessed near infrared face image.

6. A near-infrared face image generation method as claimed in claim 1, characterized in that: The step of inputting the grayscale face image to be processed into the final cyclic generative adversarial network to generate a near-infrared face image includes: The grayscale face image to be processed is input into the first generator Gx and converted into a near-infrared face image.

Images