Face feature extraction method, device, equipment and storage medium

Abstract

This application provides a method, apparatus, device, and storage medium for facial feature extraction. The image processing method includes: acquiring an image to be processed; inputting the image to be processed into a face enhancement model, performing face enhancement processing on the image to be processed through the face enhancement model to obtain an enhanced image; the image quality of the enhanced image is higher than that of the image to be processed; inputting the enhanced image into the face feature extraction model, and extracting facial features from the enhanced image through the face feature extraction model to obtain a target feature image. This application can significantly improve image quality and increase image information. When applied to the field of face recognition, due to the higher image quality, it can effectively improve the accuracy of face recognition and is better suited for face recognition applications in complex environments and scenarios.

Description

Technical Field

[0001] The present application relates to the field of image processing technology. Specifically, the present application relates to a facial feature extraction method, device, equipment and storage medium. Background Art

[0002] Existing face recognition algorithms usually adopt the deep learning models and loss functions commonly used in the industry, and are implemented through the following steps: face detection, face alignment, face feature extraction and face feature matching. The differences between different recognition methods often lie in the different loss functions.

[0003] However, no matter which loss function is used, face comparison and recognition is still based only on the information of the input image. When the input image quality is good, a higher accuracy can be achieved. However, when the input image quality is poor, such as faces in surveillance videos (low resolution, high compression, high noise, and / or motion blur), the accuracy of face recognition is often not ideal. Summary of the invention

[0004] In view of the shortcomings of the existing methods, the present application proposes a facial feature extraction method, device, equipment and storage medium to solve the technical problem that the existing face recognition algorithm has low accuracy when recognizing low-quality images.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting facial features, comprising:

[0006] Obtaining an image to be processed, where the image to be processed includes a face image;

[0007] The image to be processed is input into the face enhancement model, and the face enhancement processing is performed on the image to be processed by the face enhancement model to obtain an enhanced image; the image quality of the enhanced image is higher than the image quality of the image to be processed;

[0008] The enhanced image is input into a face feature extraction model, and the face feature extraction model is used to extract face features from the enhanced image to obtain a target feature image.

[0009] In a second aspect, an embodiment of the present application provides a facial feature extraction device, comprising:

[0010] An image acquisition module is used to acquire an image to be processed, wherein the image to be processed includes a face image;

[0011] The first image processing module is used to input the image to be processed into the face enhancement model, and perform face enhancement processing on the image to be processed by the face enhancement model to obtain an enhanced image; the image quality of the enhanced image is higher than the image quality of the image to be processed;

[0012] The second image processing module is used to input the enhanced image into a face feature extraction model, extract face features from the enhanced image through the face feature extraction model, and obtain a target feature image.

[0013] In a third aspect, an embodiment of the present application provides an image processing device, including:

[0014] Memory;

[0015] A processor, electrically connected to the memory;

[0016] The memory stores a computer program, which is executed by the processor to implement the facial feature extraction method provided in the first aspect of the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the facial feature extraction method provided in the first aspect of the embodiment of the present application.

[0018] The technical solution provided in the embodiments of the present application has at least the following beneficial effects:

[0019] The embodiment of the present application can realize enhanced processing of the image to be processed through the image enhancement model. When the image quality of the image to be processed is poor, its image quality can be greatly improved through this method; the image enhanced by the face enhancement module is subjected to feature extraction through the face feature extraction model, and the image quality of the obtained feature image is high, which is conducive to achieving more accurate face recognition and can be better applied to face recognition applications with more complex environments and / or scenes, such as surveillance videos.

[0020] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 This is a schematic diagram of face detection involved in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the distribution of key points of a human face involved in an embodiment of the present application;

[0024] Figure 3 This is a schematic diagram of the principle of the MUX layer in the embodiment of the present application;

[0025] Figure 4 This is a schematic diagram of the principle of the DEMUX layer in the embodiment of the present application;

[0026] Figure 5 A flowchart of a facial feature extraction method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the principle of a face extraction method provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of a network structure of an image enhancement model provided in an embodiment of the present application;

[0029] Figure 8 This is a schematic diagram of the network structure of the ResBlock unit in the embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the principle of training a face enhancement model and a face feature extraction model in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of a network structure of the identifier in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of a network structure of a 4-stack hourglass model in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of obtaining a key point mask in an embodiment of the present application;

[0034] Fig.13 A flowchart of another facial feature extraction method provided in an embodiment of the present application;

[0035] Fig.14 A schematic diagram of the structural framework of a facial feature extraction device provided in an embodiment of the present application;

[0036] Fig.15 A schematic diagram of the structural framework of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application is described in detail below, and examples of embodiments of the present application are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar components or components with the same or similar functions. In addition, if the detailed description of the known technology is unnecessary for the features of the present application shown, it will be omitted. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0038] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0039] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0040] First, several terms involved in this application are introduced and explained:

[0041] Face detection and key point detection technology: detect faces in face images (such as Figure 1 The key points of the face are marked in the face image (e.g. Figure 2 The facial contour, eye contour, nose, upper lip, lower lip, etc. shown in the figure); this technology is the basis of face algorithms such as face recognition and expression recognition, and can be implemented through neural networks or open source algorithms.

[0042] MUX:

[0043] The function of MUX is to combine two-dimensional matrices. It can combine four small matrices of the same size into a large matrix, where the length and width of the large matrix are both twice that of the small matrix. Figure 3 shown.

[0044] Applied in image processing, MUX can scramble the pixels of an image and combine four small images into a large image, where the length and width of the large image are both twice those of the small images. It can be used as an upsampling layer in a convolutional neural network.

[0045] DEMUX:

[0046] The function of DEMUX is to decompose a two-dimensional matrix into four small matrices whose length and width are half of the original matrix, such as Figure 4 shown.

[0047] Applied in image processing, DEMUX can scramble image pixels and decompose an image into four small images, where the length and width of the small images are both 1 / 2 of the original image. It can be used as a downsampling layer in convolutional neural networks.

[0048] Combination Figure 3 and Figure 4 It can be seen that MUX and DEMUX are inverse operations of each other, and during the processing process, only the positions of pixels are rearranged according to certain rules, and no pixel information is lost. Therefore, MUX and DEMUX can upsample and downsample images without loss.

[0049] Fighting Loss:

[0050] The adversarial loss function of the discriminator (or discriminator network, discriminator model) is:

[0051] L_D=-E x～Pdata(x) [log D(x)]-E z～Pz(z) [1-log D(G(z))] expression (1)

[0052] The adversarial loss function of the generator (or generative network, generative model) is:

[0053] L_G=E x～Pdata(x) [log D(x)]+E z～Pz(z) [1-log D(G(z))]Expression (2)

[0054] In expressions (1) and (2), D is the discriminator, G is the generator; L_D is the adversarial loss of the discriminator output, L_G is the adversarial loss of the generator output; z is the input of the generator, Pz is the input set of the generator; x is a true sample, Pdata is the true sample set, that is, the set that can make the discriminator output 1; D(x) is the result obtained after the true sample is input to the discriminator, G(z) is the output result of the generator; E x～Pdata(x) Indicates that x is sampled from the true sample set, and subsequent operations are performed on any x; E z～Pz(z) It means that z is sampled from the input set Pz of the generator and subsequent operations are performed on any z.

[0055] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0056] The present application embodiment provides a method for extracting facial features. Figure 5 As shown, the method includes:

[0057] S501: Acquire at least one image to be processed.

[0058] The images to be processed include face images.

[0059] S502, inputting the image to be processed into a face enhancement model, performing face enhancement processing on the image to be processed through the face enhancement model to obtain an enhanced image.

[0060] The image quality of the enhanced image is higher than that of the image to be processed.

[0061] S503, inputting the enhanced image into a face feature extraction model, extracting face features from the enhanced image through the face feature extraction model, and obtaining a target feature image.

[0062] The image quality in the embodiment of the present application includes at least one parameter of the image resolution, noise, compression degree and blur degree. The image quality of image A is higher than the image quality of image B, which means that image A has at least one of the following parameters compared with image B: the resolution of image A is greater than the resolution of image B, the noise of image A is less than the noise of image B, the compression degree of image A is less than the compression degree of image B, and the blur degree of image A is less than the blur degree of image B.

[0063] The image processing method provided in the embodiment of the present application has at least the following beneficial effects:

[0064] The embodiment of the present application can realize enhanced processing of the image to be processed through the image enhancement model. When the image quality of the image to be processed is poor, its image quality can be greatly improved through this method; the image enhanced by the face enhancement module is subjected to feature extraction through the face feature extraction model, and the image quality of the obtained feature image is high, which is conducive to achieving more accurate face recognition and can be better applied to face recognition applications with more complex environments and / or scenes, such as surveillance videos.

[0065] Optionally, the image to be processed in the embodiment of the present application is obtained by performing face detection, face key point detection and face alignment on the original image.

[0066] The original image can be a single image or an image of a frame captured from a video.

[0067] When performing face detection and face key point detection, it can be implemented based on the aforementioned face detection and key point detection technology. In one example, the function interface of dlib can be called to detect key points in a face image (referred to as face key points). The number of detectable key points is usually 68, such as Figure 2 shown.

[0068] For the original image obtained from the video, since the size, position, angle, etc. of the face in the original image are all highly random, in order to make the face in the original image accurately move according to the facial features in the video, it is necessary to align the face according to the position of the facial features, which specifically includes the following steps:

[0069] 1) Face angle alignment

[0070] If the face detected in the detection frame is directly aligned in angle and rotated, the obtained face may be incomplete. Therefore, the face needs to be extended outward by a certain number of pixels to obtain an extended image (expressed as I extend) so that the face is complete after rotation.

[0071] Assume that the detected face frame is D, with a width of d_w and a height of d_h. Expand outward along the face frame D, specifically, expand by extend_w=2*d_w pixels in the width direction and extend_h=2*d_h pixels in the height direction.

[0072] The extended face frame has a new range. Assume that the coordinates of the upper left corner of the extended face frame are (new_left, new_top) and the coordinates of the lower right corner are (new_right, new_bottom). The image captured in this range is I extend.

[0073] When performing face angle alignment, the left eye coordinates and the right eye coordinates are used to calculate the angle. Specifically, the calculation is as follows: Figure 2 The angle angle between the line connecting 37 points (left corner of the left eye) and 46 points (right corner of the right eye) of the 68 key points shown and the horizontal direction is taken as the rotation center (the coordinates (new_left, new_top) need to be subtracted here), and the transformation matrix H is obtained with angle as the angle. The image I extend is transformed according to H to obtain Iextend_rotation.

[0074] The width, height, left and right in the embodiments of the present application are based on the positional relationship shown in the accompanying drawings. Figure 1 The horizontal direction in the middle and the height direction in the Figure 1 The vertical direction in the left eye Figure 2 The left eye area in the figure, the right eye is Figure 2 The eye area on the middle right.

[0075] 2) Transformation of facial key point positions

[0076] Assume that the position of a certain facial key point is The position of the new key point after the key point is transformed is P_new=H*P.

[0077] 3) Human eye center alignment

[0078] In Iextend_rotation, we get the new key points of the face, especially Figure 2 The 4 positions shown, 37 o'clock, 40 o'clock, 43 o'clock and 46 o'clock, correspond to Figure 2 The four eye corners of the left eye, the left eye, the right eye, the left eye, and the right eye are aligned so that the eyes are horizontal. The middle position LM of the left eye can be calculated based on points 37 and 40, and the middle position RM of the right eye can be calculated based on points 43 and 46.

[0079] The distance between LM and RM is used as the standard for expansion. Specifically, the distance D between LM and RM is determined, referring to Figure 6 The face extraction method shown in the figure determines the face. Taking LM as the base point, it extends 1.5D to the left and 2D upward. Taking RM as the base point, it extends 1.5D to the right and 2D upward. The following is obtained by this operation: Figure 6 The range of the outer frame shown is the final face.

[0080] Optionally, before inputting the image to be processed into the face enhancement model, the embodiment of the present application further includes: scaling the image to be processed to obtain images to be processed of multiple target sizes.

[0081] The number and specific values ​​of the target sizes can be determined according to actual needs. The image to be processed can be resized in any of the following ways to obtain images of multiple sizes:

[0082] Method 1: scaling the image to be processed to images of multiple target sizes. The specific scaling method may be any one of linear interpolation, bilinear interpolation, bicubic interpolation, nearest neighbor interpolation, etc.

[0083] In an example, if the multiple target sizes include 32×32, 64×64, 128×128, and 256×256, the image to be processed can be directly scaled to the four sizes by any method such as linear interpolation, bilinear interpolation, bicubic interpolation, and nearest neighbor interpolation.

[0084] Method 2: determine the size range to which the image to be processed belongs, scale the image to be processed to the target size within the range to be enhanced, and obtain the image to be processed of the target size; scale the image to be processed of the target size to the target size within at least one other size range, and obtain the image to be processed of the target size within the size range.

[0085] The size range and the target size within the size range can be pre-divided and determined according to actual needs. In one example, the size range of the image to be processed is determined. If the image to be processed is less than or equal to 48×48, the image to be processed is scaled to 32×32; if the image to be processed is larger than 48×48 and not larger than 96×96, the image to be processed is scaled to 64×64; if the image to be processed is larger than 96×96 and not larger than 192×192, the image to be processed is scaled to 128×128; if the image to be processed is larger than 192×192, the image to be processed is scaled to 256×256; the image obtained by the above scaling can be further scaled to other sizes. For example, after scaling the image to be processed to 32×32, the 32×32 image can be further scaled to three other target sizes.

[0086] In the embodiment of the present application, before performing enhancement processing on the image to be processed, the image to be processed can be resized in advance, and the obtained multi-size image can be input into the face enhancement model. The image enhancement model performs enhancement processing on the multi-size image. Compared with the enhancement processing of a single-size image with a single input, the image quality can be improved by utilizing multiple levels of image information corresponding to the multi-size image.

[0087] Optionally, in step S502, the image to be processed is input into a face enhancement model, and the image to be processed is subjected to face enhancement processing by the face enhancement model to obtain an enhanced image, including:

[0088] The images to be processed of multiple sizes are respectively input into the face enhancement model through multiple input terminals of the face enhancement model, and the face enhancement model is used to perform face enhancement processing on the images to be processed of multiple sizes to obtain multiple enhanced images.

[0089] Optionally, in step S503, the enhanced image is input into a face feature extraction model, and features are extracted from the enhanced image by the face feature extraction model to obtain a target feature image, including:

[0090] The multiple enhanced images are respectively input into the face feature extraction model through the multiple input terminals of the face feature extraction model, and the face feature extraction model is used to extract features from the multiple enhanced images to obtain a target feature image.

[0091] Optionally, the face enhancement model in the implementation of the present application includes multiple cascaded face enhancement modules.

[0092] For N target sizes of images to be processed, the images to be processed of the multiple target sizes are respectively input into the face enhancement model through the multiple input terminals of the face enhancement model, and the face enhancement model is used to perform face enhancement processing on the images to be processed of the multiple target sizes to obtain multiple enhanced images, including:

[0093] The image to be processed of the first target size is input into the first-level face enhancement module through the first input end of the face enhancement model, and the first-level face enhancement module performs face enhancement processing on the image to be processed of the first target size, and outputs a first enhanced image; the image to be processed of the i-th target size is input through the i-th input end of the face enhancement model, and is spliced ​​with the i-1th enhanced image output by the i-1th level face enhancement module to obtain a spliced ​​enhanced image, and the spliced ​​enhanced image is input into the i-th level face processing module, and the i-th level face processing module performs face enhancement processing on the spliced ​​enhanced image, and outputs the i-th enhanced image; the i-th target size is larger than the i-1th target size, and i is an integer greater than 1 and not greater than N.

[0094] Figure 7 A face enhancement model and a face feature extraction model are shown, and the face enhancement model includes four cascaded face enhancement modules 711-714. The first-level image enhancement module 711 includes an input end, namely input end 1; the second-level image enhancement module 712 includes two input ends, specifically one end connected to the output end of the first-level image enhancement module 711, and input end 2; the third-level image enhancement module 713 includes two input ends, specifically one end connected to the output end of the second-level image enhancement module 712, and input end 3; the fourth-level image enhancement module 714 includes two channels, specifically one end connected to the output end of the third-level image enhancement module 713, and input end 4. For an image enhancement module including two input ends, the images inputted at the two input ends can be inputted into the image enhancement module of this level after the channels are spliced, namely the splicing of the two input ends.

[0095] If the size of the image obtained by scaling the image to be processed is 32×32, 64×64, 128×128, and 256×256, refer to Figure 7 In the example shown, a 32×32 image can be input from input terminal 1 to the first-level image enhancement module 711, a 64×64 image can be input from input terminal 2 and input into the second-level face enhancement module 712 after being spliced ​​with the enhanced image output by the first-level face enhancement module 711, a 128×128 image can be input from input terminal 3 and input into the third-level face enhancement module 713 after being spliced ​​with the enhanced image output by the second-level face enhancement module 712, and a 256×256 image can be input from input terminal 4 and input into the fourth-level face enhancement module 714 after being spliced ​​with the enhanced image output by the third-level face enhancement module 713.

[0096] The embodiment of the present application can realize multi-level processing of input images by setting a multi-level face enhancement module in the face enhancement model, thereby gradually improving the quality of the image. At the same time, face enhancement modules at different levels can input images of different sizes, thereby realizing the input and stitching of multi-sized images and obtaining more comprehensive image information.

[0097] Optionally, in the N-level face enhancement module, each level of the face enhancement module from the first level to the N-1th level includes a cascaded residual unit and an upsampling unit, and the N-th level face enhancement module includes a residual unit.

[0098] The residual unit is used to extract features of the image input to the unit; the upsampling unit is used to upsample the image input to the unit.

[0099] In such Figure 7 In the example shown, the first to third image enhancement modules 711-713 each include two cascaded ResBlock units (as residual units) and Mux (as upsampling units), and the last image enhancement module 714 includes one ResBlock unit (as residual unit).

[0100] In an optional embodiment, the ResBlock unit includes Figure 8 As shown, there are two convolutional layers (Conv) and two activation layers (ReLu), the convolutional layers and the activation layers are arranged alternately, and the input and output of the residual unit can be connected across layers; in actual use, the number of convolutional layers and the number of activation layers can be set according to actual needs, and the activation function of the activation layer can also be other functions, such as the sigmoid function.

[0101] Optionally, the facial feature extraction model in the embodiment of the present application includes N cascaded facial feature extraction modules and a fully connected layer cascaded after the Nth level facial feature extraction module.

[0102] Inputting the multiple enhanced images into the face feature extraction model through the multiple input terminals of the face feature extraction model, extracting features from the multiple enhanced images through the face feature extraction model, and obtaining a target feature image, including:

[0103] Input the Nth enhanced image output by the Nth level face enhancement module and the N-1th enhanced image output by the N-1th level face enhancement module through the Nth input terminal and the N-1th input terminal of the face feature extraction model respectively, and splice them to obtain a spliced ​​feature image, input the spliced ​​feature image into the first level face feature extraction module, perform feature extraction on the Nth enhanced image through the first level face feature extraction module, and output a first feature image;

[0104] The j-th enhanced image output by the j-th level face enhancement module is input through the j-th input terminal of the face feature extraction model, and is spliced ​​with the j-1-th feature image output by the j-1-th level face feature extraction module to obtain a spliced ​​feature image, and the spliced ​​feature image is input into the j-th level face feature extraction module, and the j-th level face feature extraction module performs feature extraction on the spliced ​​feature image to output the j-th feature image;

[0105] Inputting the N-1th feature image output by the N-1th level face feature extraction module into the N-1th level face feature extraction module, performing feature extraction on the N-1th feature image through the N-1th level face feature extraction module, and outputting a target feature image;

[0106] j is an integer greater than 1 and less than N.

[0107] exist Figure 7 In the example shown, the face feature extraction model includes four cascaded face feature extraction modules 721-724 and an FC layer (i.e., fully connected layer) cascaded after the fourth level face feature extraction module 724. The enhanced images output by each level of face enhancement module can be input into each face feature extraction module through different input terminals.

[0108] Specifically, Figure 7 As shown, the enhanced image output by the last-level face enhancement module 714 and the enhanced image output by the third-level face enhancement module 713 are respectively input through two input terminals and spliced ​​to input into the first-level face feature extraction module 721; the enhanced image output by the second-level face enhancement module 712 is input through the third input terminal and spliced ​​with the feature image output by the first-level face feature extraction module 721 to input into the second-level face feature extraction module 722; the enhanced image output by the first-level face enhancement module 711 is input through the fourth input terminal and spliced ​​with the feature image output by the second-level face feature extraction module 722 to input into the third-level face feature extraction module 723.

[0109] The embodiment of the present application sets up a multi-level face feature extraction module in the face feature extraction model, which can realize multi-level processing of the input image (i.e., enhanced image). At the same time, different levels of face feature extraction modules can input enhanced images of different sizes (i.e., outputs of different face enhancement modules), which can realize the input and splicing of multi-size images and make full use of the feature information in the enhancement process to improve the accuracy of feature extraction.

[0110] Optionally, in an N-level facial feature extraction module, each level of the facial feature extraction module from the first level to the N-1th level includes a cascaded residual unit and a downsampling unit, and the N-level facial feature extraction module includes a residual unit; the residual unit is used to extract features of an image input into the unit; and the residual unit sub-module is used to downsample the image input into the unit.

[0111] exist Figure 7 In the example shown, the first to third level facial feature extraction modules 721-723 each include two cascaded ResBlock units (as residual units) and Mux units (as upsampling units), and the last level facial feature extraction module 724 includes one ResBlock unit (as a residual unit).

[0112] In an optional embodiment, the structure of the ResBlock unit is as follows Fig. 9 As shown, in actual use, the number of convolutional layers and the number of activation layers can be set according to actual needs, and the activation function of the activation layer can also be other functions, such as the sigmoid function.

[0113] The facial feature extraction model in the embodiment of the present application performs feature extraction on the image enhanced by the face enhancement module. The images output by each level of the face enhancement module are input into the facial feature extraction model. The facial feature extraction model can make full use of the feature information in the enhancement process when extracting features. The obtained feature image has a high image quality, which is conducive to achieving more accurate face recognition.

[0114] Alternatively, if Fig. 9 As shown, the face enhancement model in the embodiment of the present application is trained in the following way:

[0115] Acquire training sample data; perform alternate iterative training on the initial model combination and the initial discriminator based on the training sample data until the loss function value of the trained model combination and the loss function value of the discriminator both meet preset convergence conditions; the initial model combination includes: an initial face enhancement model and an initial face feature extraction model.

[0116] The training sample data includes multiple groups of first sample images and second sample images with the same image content (the first sample images in each group have the same image content as the second sample images in the same group), and the image quality of the first sample images in each group of sample images is lower than that of the second sample images.

[0117] In an example, the resolution range of the first sample image is [32×32, 256×256), and the resolution of the second sample image is 256×256.

[0118] Optionally, refer to Fig. 9 In the stage of training the initial discriminator, for each group of sample images, the first sample image is input into the current face enhancement model, and the first sample image is subjected to face enhancement processing by the current face enhancement model to obtain a sample enhanced image; the sample enhanced image and the second sample image are input into the discriminator, and through identification by the discriminator, a first identification result of the sample enhanced image and an identification result of the second sample image are obtained; according to the first identification result of the sample enhanced image and the identification result of the second sample image, a first generative adversarial loss is determined; according to the first generative adversarial loss, the parameters of the initial discriminator are adjusted to obtain the discriminator after the parameter adjustment.

[0119] The current face enhancement model in the embodiments of the present application is the initial face enhancement model or the face enhancement model after the previous parameter adjustment; for the first group of sample images participating in the alternating iterative training, the current face enhancement model is the initial face enhancement model, and for the second group and subsequent sample images participating in the alternating iterative training, the current face enhancement model is the face enhancement model after the previous parameter adjustment.

[0120] In an optional embodiment, during the stage of training the discriminator, before inputting the first sample image into the current face enhancement model, the first sample image may be scaled to obtain multiple first sample images of target sizes, and then the multiple first sample images of target sizes may be input into the face enhancement model.

[0121] In one example, if the face image enhancement model includes multiple cascaded face image enhancement modules, the enhanced image input to the discriminator is the enhanced image output by the last level face image enhancement module.

[0122] The first generative adversarial loss can be calculated according to the aforementioned expression (1), and the second sample image is the true sample (Ground Truth) x in expression (1).

[0123] In an optional embodiment, the discriminator in the embodiment of the present application includes multiple convolutional layers (conv), pooling layers (Pool), fully connected layers (FC) and activation layers (sigmoid), and the cascade mode of each layer is as follows: Fig.10 As shown in the figure, each time the feature image output by the convolutional layer undergoes a pooling layer, the size of the feature image is reduced by half. After being processed by the last convolutional layer, it passes through a fully connected layer and an activation layer to output a scalar value between 0 and 1, where 1 represents true and 0 represents false.

[0124] Optionally, refer to Fig. 9 In the stage of training the initial model combination, for each group of sample images, the sample enhanced image and the second sample image obtained based on the first sample image are respectively input into the current face feature extraction model, and the face features of the sample enhanced image and the second sample image are respectively extracted by the current face feature extraction model to obtain the extracted first sample feature vector and the second sample feature vector; the sample enhanced image is input into the discriminator based on the previous parameter adjustment, and the second identification result of the sample enhanced image is obtained through the identification of the discriminator after the previous parameter adjustment; the total image loss of the model combination is determined according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image; the parameters of the current face enhancement model and the current face feature extraction model are adjusted according to the total image loss to obtain the face enhancement model and the face feature extraction model after this parameter adjustment.

[0125] The current facial feature extraction model in the embodiments of the present application is the initial facial feature extraction model or the facial feature extraction model after the previous parameter adjustment; for the first group of sample images participating in the alternating iterative training, the current facial feature extraction model is the initial facial feature extraction model, and for the second group and subsequent sample images participating in the alternating iterative training, the current facial feature extraction model is the facial feature extraction model after the previous parameter adjustment.

[0126] The embodiment of the present application is based on paired sample image data and adopts a method of alternating iterative training of the initial model combination and the initial discriminator to achieve joint training of the initial face enhancement model and the initial face feature extraction model. More accurate face enhancement models and face feature extraction models can be trained to improve the quality of face feature extraction.

[0127] Optionally, the total image loss in the embodiment of the present application includes: feature loss, pixel loss and a second generative adversarial loss.

[0128] Determining a total image loss of the model combination according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image, including:

[0129] Determine a feature loss of the first sample feature vector relative to the second sample feature vector; determine a pixel loss of each pixel of the sample enhanced image relative to each pixel of the second sample image; and determine a second generative adversarial loss based on the second identification result.

[0130] In one example, the feature loss of a sample feature vector can be calculated as follows:

[0131] L Vec_MSE =MSE(V output ,V GroundTruth ) Expression (3)

[0132] In expression (3), L Vec_MSE is the feature loss of the first sample feature vector relative to the second sample feature vector, V output is the first sample feature vector output by the face feature extraction model based on the enhanced image, V GroundTruth is the second feature vector output by the face feature extraction model based on the second sample image; MSE (Mean Square Error) represents the calculation of V output and V GroundTruth The mean square error of .

[0133] The pixel loss in the embodiment of the present application includes any one of L1 loss (also called primary loss) and L2 loss (also called secondary loss).

[0134] In one example, the L1 loss can be calculated as follows:

[0135] L MSE =MSE(I output ,I GroundTruth ) Expression (4)

[0136] In expression (4), L MSE To enhance the pixel loss of all pixels in the image relative to all pixels in the second sample image, I output is the pixel value of all pixels in the enhanced image output by the face enhancement model (such as the enhanced image output by the last level face enhancement module in the multi-level face enhancement module), I GroundTruth is the pixel value of all pixels in the second sample image; MSE represents the calculation I output and I GroundTruth The mean square error of .

[0137] In another example, the L2 loss can be calculated as follows:

[0138] L MAE =MAE(I output ,I GroundTruth ) Expression (5)

[0139] In expression (4), L MAE To enhance the pixel loss of all pixels in the image relative to all pixels in the second sample image, I output is the pixel value of all pixels in the enhanced image output by the face enhancement model (such as the enhanced image output by the last level face enhancement module in the multi-level face enhancement module), I GroundTruth is the pixel value of all pixels in the second sample image; MAE (Mean Absolute Error) represents the average absolute error of the calculated I output and I GroundTruth The mean absolute error.

[0140] The second generative adversarial loss in the embodiment of the present application can be calculated according to the aforementioned expression (2), and the second sample image is the true sample (Ground Truth) x in expression (1).

[0141] In an embodiment of the present application, the parameters of the current face enhancement model and the current face feature extraction model can be adjusted separately according to the feature loss, pixel loss and the second generative adversarial loss, or the parameters of the current face enhancement model and the current face feature extraction model can be adjusted according to the total loss after the three losses are superimposed. When the three losses are superimposed, a corresponding weight can be set for each loss, and the superposition is performed based on the weight.

[0142] Training is performed based on three types of losses: feature loss, pixel loss, and second generative adversarial loss. More accurate face enhancement models and face feature extraction models can be trained, thereby improving the face enhancement processing capability of the face enhancement model and the face feature extraction capability of the face feature extraction model.

[0143] Optionally, the total image loss in the embodiment of the present application also includes: similarity loss.

[0144] Determining the total image loss of the model combination according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image also includes: determining the similarity loss of the sample enhanced image relative to the second sample image.

[0145] In an embodiment of the present application, the parameters of the current face enhancement model and the current face feature extraction model can be adjusted based on the similarity loss alone, or the similarity loss can be superimposed on the previous three losses, and the parameters of the current face enhancement model and the current face feature extraction model can be adjusted based on the total loss after superposition. When superimposing the four losses, a corresponding weight can be set for each loss, and the superposition is performed based on the weight.

[0146] Training is carried out based on four types of losses: feature loss, pixel loss, second generation adversarial loss, and similarity loss. More accurate face enhancement models and face feature extraction models can be trained, further improving the face enhancement processing capability of the face enhancement model and the face feature extraction capability of the face feature extraction model.

[0147] Optionally, determining a similarity loss of the sample enhanced image relative to the second sample image includes:

[0148] The sample enhanced image and the second sample image are respectively input into the key point detection module to obtain the key point heat map of the sample enhanced image and the key point heat map of the second sample image output by the key point detection module; the similarity loss of the key point heat map of the sample enhanced image relative to the key point heat map of the sample image is determined as the similarity loss of the sample enhanced image relative to the sample image.

[0149] Optionally, the key point detection module includes a 4-stack hourglass model. The key point heat map output by the 4-stack hourglass model shows the probability that a certain pixel in the image is a facial landmark. For an image with 68 facial landmarks, 68 key point heat maps can be generated.

[0150] In the embodiment of the present application, the 4-stack hourglass model includes multiple ResBlocks, pooling layers and anti-pooling layers, and the arrangement of each layer is as follows Fig.11 As shown,

[0151] Optionally, refer to Fig.12 The specific process of determining the similarity loss based on the key point heat map is as follows: Taking the key point heat map of 68 facial key points as an example, the 68 key point heat maps (such as Fig.11 The key points in the key point heat map (Landmarks) shown on the far left are merged according to the facial parts (such as the left eye, right eye, nose, mouth, and contour) to obtain the key point heat map of each facial part, as shown in Figure 1. Fig.12 The five key point heat maps of the five facial components shown in the middle part are then processed by the softmax logistic regression model to obtain the following: Fig.12 The five key point masks (Mask) shown on the far right; calculate the similarity loss of the five key point masks of the first enhanced image relative to the five key masks of the sample image.

[0152] This method can finely divide the original 68 facial key points according to specific facial parts, and calculate the similarity loss based on the finely divided facial parts. The calculated similarity loss has a high accuracy.

[0153] In one example, the similarity loss of the keypoint mask can be calculated as follows:

[0154] L a =MSE(A output ,A GroundTruth ) Expression (6)

[0155] In expression (5), L a To enhance the similarity loss of the keypoint mask of the image relative to the keypoint mask of the second sample image, A output is the pixel value of the key point mask of the enhanced image of the face enhancement model (such as the enhanced image output by the last level face enhancement module in the multi-level face enhancement module), A GroundTruth is the pixel value of the key point mask of the second sample image; MSE represents the calculation of A output and A GroundTruth The mean square error of .

[0156] Alternatively, if Fig.13 As shown, the image processing method provided in the embodiment of the present application further includes the following steps:

[0157] S504: When there are more than two images to be processed, feature matching is performed on feature vectors corresponding to any two images to be processed to determine whether the two images to be processed belong to the same photographed object.

[0158] If the two images to be processed for feature matching are both face images, feature matching is performed on the two images to determine whether the two images to be processed belong to the same face, thereby realizing face recognition.

[0159] In an optional implementation, feature matching of two images to be processed may be performed by calculating the Euclidean distance between two feature vectors, as follows:

[0160]

[0161] In expression (7), x is the feature vector obtained after the first image to be processed is enhanced by the face enhancement model and extracted by the face feature extraction model, including n elements, x i is the i-th element in the feature vector x; y is the feature vector obtained after the second image to be processed is enhanced by the face enhancement model and extracted by the face feature extraction model, including n elements, y i is the i-th element in the eigenvector y; d(x,y) is the Euclidean distance between the eigenvector x and the eigenvector y; n is a positive integer, and i is a positive integer not greater than n.

[0162] If the Euclidean distance is less than or equal to a preset distance threshold, it can be considered that the two images to be processed belong to the same face; otherwise, it is considered that the two images to be processed do not belong to the same face.

[0163] The embodiment of the present application performs face matching based on the feature vector obtained after the enhanced image output by the face enhancement model is extracted by the face feature extraction model, which can improve the accuracy of face matching and further improve the accuracy of face recognition.

[0164] Based on the same inventive concept, the embodiment of the present application provides a facial feature extraction device for executing any one of the image processing methods provided in the embodiment of the present application. Fig.14 As shown, the facial feature extraction device 1400 includes: an image acquisition module 1401, a first image processing module 1402 and a second image processing module 1403.

[0165] An image acquisition module 1401 is used to acquire an image to be processed, wherein the image to be processed includes a face image;

[0166] The first image processing module 1402 is used to input the image to be processed into the face enhancement model, and perform face enhancement processing on the image to be processed through the face enhancement model to obtain an enhanced image; the image quality of the enhanced image is higher than the image quality of the image to be processed;

[0167] The second image processing module 1403 is used to input the enhanced image into a face feature extraction model, extract face features from the enhanced image through the face feature extraction model, and obtain a target feature image.

[0168] The specific structures of the face enhancement model and the face feature extraction model can refer to the contents of the aforementioned method embodiments and will not be repeated here.

[0169] Optionally, the facial feature extraction device 1400 provided by the present application further includes: an image scaling module.

[0170] The image scaling module is used to scale the image to be processed before inputting it into the face enhancement model to obtain images to be processed of multiple target sizes.

[0171] Optionally, the first image processing module 1402 is specifically used to: input the images to be processed of multiple target sizes into the face enhancement model through multiple input ends of the face enhancement model respectively, and perform face enhancement processing on the images to be processed of multiple target sizes through the face enhancement model to obtain multiple enhanced images.

[0172] Optionally, for N to-be-processed images of target sizes, the first image processing module 1402 is specifically configured to:

[0173] The image to be processed of the first target size is input into the first-level face enhancement module through the first input end of the face enhancement model, and the first-level face enhancement module performs face enhancement processing on the image to be processed of the first target size, and outputs a first enhanced image; the image to be processed of the i-th target size is input through the i-th input end of the face enhancement model, and is spliced ​​with the i-1th enhanced image output by the i-1th level face enhancement module to obtain a spliced ​​enhanced image, and the spliced ​​enhanced image is input into the i-th level face processing module, and the i-th level face processing module performs face enhancement processing on the spliced ​​enhanced image, and outputs the i-th enhanced image; the i-th target size is larger than the i-1th target size, and i is an integer greater than 1 and not greater than N.

[0174] Optionally, the second image processing module 1403 is specifically used to: input the multiple enhanced images into the face feature extraction model through the multiple input terminals of the face feature extraction model respectively, perform feature extraction on the multiple enhanced images through the face feature extraction model, and obtain a target feature image.

[0175] Optionally, the second image processing module 1403 is specifically used to: input the Nth enhanced image output by the Nth level face enhancement module and the N-1th enhanced image output by the N-1th level face enhancement module, respectively, through the Nth input end and the N-1th input end of the face feature extraction model and splice them to obtain a spliced ​​feature image, input the spliced ​​feature image into the first level face feature extraction module, perform feature extraction on the Nth enhanced image through the first level face feature extraction module, and output the first feature image; input the jth enhanced image output by the jth level face enhancement module through the face feature extraction model The j-th feature image inputted at the j-th input terminal and outputted from the j-1-th level face feature extraction module is spliced ​​to obtain a spliced ​​feature image, the spliced ​​feature image is inputted into the j-th level face feature extraction module, the spliced ​​feature image is subjected to feature extraction by the j-th level face feature extraction module, and the j-th feature image is outputted; the N-1-th feature image outputted from the N-1-th level face feature extraction module is inputted into the N-th level face feature extraction module, the N-1-th feature image is subjected to feature extraction by the N-th level face feature extraction module, and the target feature image is outputted; j is an integer greater than 1 and less than N.

[0176] Optionally, the facial feature extraction device 1400 provided by the present application further includes: a model training module.

[0177] The model training module is used to: obtain training sample data; perform alternating iterative training on the initial model combination and the initial discriminator based on the training sample data until the loss function value of the trained model combination and the loss function value of the discriminator both meet the preset convergence conditions; the initial model combination includes: an initial face enhancement model and an initial face feature extraction model.

[0178] Optionally, the model training module is specifically used for: in the stage of training the initial discriminator, for each group of sample images, inputting the first sample image into the current face enhancement model, performing face enhancement processing on the first sample image through the current face enhancement model to obtain a sample enhanced image; the current face enhancement model is the initial face enhancement model or the face enhancement model after the previous parameter adjustment; inputting the sample enhanced image and the second sample image into the discriminator, and obtaining a first identification result of the sample enhanced image and an identification result of the second sample image through identification by the discriminator; determining a first generative adversarial loss based on the first identification result of the sample enhanced image and the identification result of the second sample image; and adjusting the parameters of the initial discriminator based on the first generative adversarial loss to obtain the discriminator after this parameter adjustment.

[0179] Optionally, the model training module is also used for: in the stage of training the initial model combination, for each group of sample images, the sample enhanced image and the second sample image obtained based on the first sample image are respectively input into the current face feature extraction model, and the face features of the sample enhanced image and the second sample image are respectively extracted by the current face feature extraction model to obtain the extracted first sample feature vector and the second sample feature vector; the current face feature extraction model is the initial face feature extraction model or the face feature extraction model after the previous parameter adjustment; the sample enhanced image is input into the discriminator based on the previous parameter adjustment, and the second identification result of the sample enhanced image is obtained through identification by the discriminator after the previous parameter adjustment; the total image loss of the model combination is determined according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image; the parameters of the current face enhancement model and the current face feature extraction model are adjusted according to the total image loss to obtain the face enhancement model and the face feature extraction model after this parameter adjustment.

[0180] Optionally, when determining the total image loss, the model training module is specifically used to: determine the feature loss of the first sample feature vector relative to the second sample feature vector; determine the pixel loss of each pixel of the sample enhanced image relative to each pixel of the second sample image; and determine the second generative adversarial loss based on the second identification result.

[0181] Optionally, when determining the total image loss, the model training module is further used to: determine the similarity loss of the sample enhanced image relative to the second sample image.

[0182] Optionally, when determining the similarity loss, the model training module is specifically used to: input the sample enhanced image and the second sample image into the key point detection module respectively, to obtain the key point heat map of the sample enhanced image and the key point heat map of the second sample image output by the key point detection module; determine the similarity loss of the key point heat map of the sample enhanced image relative to the key point heat map of the sample image, as the similarity loss of the sample enhanced image relative to the sample image.

[0183] The facial feature extraction device 1400 of this embodiment can execute any facial feature extraction method provided in the embodiments of the present application, and the implementation principles are similar. The contents not shown in detail in this embodiment can be referred to the previous method embodiments and will not be repeated here.

[0184] Based on the same inventive concept, an embodiment of the present application provides a facial feature extraction device, which includes: a memory and a processor, and the memory is electrically connected to the processor.

[0185] A computer program is stored in the memory, and the computer program is executed by the processor to implement any facial feature extraction method provided in the embodiments of the present application.

[0186] Those skilled in the art will appreciate that the facial feature extraction device provided in the embodiments of the present application may be specially designed and manufactured for the desired purpose, or may also include known devices in a general-purpose computer. These devices have computer programs stored therein, which are selectively activated or reconstructed. Such computer programs may be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, respectively.

[0187] In an optional embodiment, the present application provides a facial feature extraction device, such as Fig.15 As shown, the image processing device 1500 includes: a memory 1501 and a processor 1502 , and the memory 1501 and the processor 1502 are electrically connected, such as connected via a bus 1503 .

[0188] Optionally, the memory 1501 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 1502. The processor 1502 is used to execute the application code stored in the memory 1501 to implement any facial feature extraction method provided in the embodiments of the present application.

[0189] The memory 1501 may be a ROM (Read-Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0190] Processor 1502 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1902 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0191] The bus 1503 may include a path to transmit information between the above components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0192] Optionally, the image processing device 1500 may further include a transceiver 1504. The transceiver 1504 may be used for receiving and sending signals. The transceiver 1504 may allow the electronic device 1500 to communicate with other devices wirelessly or by wire to exchange data. It should be noted that in actual applications, the transceiver 1504 is not limited to one.

[0193] Optionally, the image processing device 1500 may further include an input unit 1505. The input unit 1505 may be used to receive input numbers, characters, images and / or sound information, or generate key signal input related to user settings and function control of the electronic device 1500. The input unit 1505 may include, but is not limited to, one or more of a touch screen, a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, a camera, a microphone, etc.

[0194] Optionally, the image processing device 1500 may further include an output unit 1506. The output unit 1506 may be used to output or display information processed by the processor 1502. The output unit 1506 may include but is not limited to one or more of a display device, a speaker, a vibration device, and the like.

[0195] Although Fig.15 The image processing apparatus 1500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0196] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any image processing method provided in the embodiment of the present application is implemented.

[0197] The computer readable medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM, RAM, EPROM (Erasable Programmable Read-Only Memory), EEPROM, flash memory, magnetic card or optical card. In other words, the readable medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0198] The embodiment of the present application provides a computer-readable storage medium applicable to any of the above-mentioned image processing methods, which will not be described in detail here.

[0199] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0200] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0201] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0202] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for extracting facial features, characterized in that: include: Acquire an image to be processed, wherein the image to be processed includes a face image; Inputting the image to be processed into a face enhancement model, performing face enhancement processing on the image to be processed through the face enhancement model to obtain an enhanced image; the image quality of the enhanced image is higher than the image quality of the image to be processed; Inputting the enhanced image into a face feature extraction model, extracting face features from the enhanced image through the face feature extraction model, and obtaining a target feature image; Before inputting the image to be processed into the face enhancement model, the method further includes: Scaling the size of the image to be processed to obtain the image to be processed of multiple target sizes; Inputting the image to be processed into the face enhancement model, performing face enhancement processing on the image to be processed through the face enhancement model to obtain an enhanced image, including: Inputting the to-be-processed images of multiple target sizes into the face enhancement model through multiple input terminals of the face enhancement model respectively, and performing face enhancement processing on the to-be-processed images of multiple target sizes through the face enhancement model to obtain multiple enhanced images; The enhanced image is input into the face feature extraction model, and features are extracted from the enhanced image by the face feature extraction model to obtain a target feature image, including: Inputting the plurality of enhanced images into the face feature extraction model through the plurality of input terminals of the face feature extraction model respectively, performing feature extraction on the plurality of enhanced images through the face feature extraction model to obtain a target feature image; The face enhancement model includes N cascaded face enhancement modules; For the to-be-processed images of N target sizes, the to-be-processed images of the multiple target sizes are respectively input into the face enhancement model through the multiple input terminals of the face enhancement model, and the to-be-processed images of the multiple target sizes are subjected to face enhancement processing by the face enhancement model to obtain multiple enhanced images, including: Inputting the to-be-processed image of the first target size into the first-level face enhancement module through the first input terminal of the face enhancement model, performing face enhancement processing on the to-be-processed image of the first target size through the first-level face enhancement module, and outputting a first enhanced image; Input the image to be processed of the i-th target size through the i-th input terminal of the face enhancement model, and splice it with the i-1th enhanced image output by the i-1th level face enhancement module to obtain a spliced ​​enhanced image, input the spliced ​​enhanced image into the i-th level face processing module, perform face enhancement processing on the spliced ​​enhanced image through the i-th level face processing module, and output the i-th enhanced image; The i-th target size is greater than the (i-1)-th target size, where i is an integer greater than 1 and not greater than N.

2. The facial feature extraction method according to claim 1, characterized in that: The face enhancement module of each level from the first level to the N-1 level includes a cascaded residual unit and an upsampling unit, and the face enhancement module of the Nth level includes a residual unit; The residual unit is used to extract features from an image input into the unit; The up-sampling unit is used to perform up-sampling processing on the image input into the unit.

3. The facial feature extraction method according to claim 1, characterized in that: The facial feature extraction model includes N cascaded facial feature extraction modules and a fully connected layer cascaded after the Nth level of facial feature extraction modules; Inputting the plurality of enhanced images into the face feature extraction model through the plurality of input terminals of the face feature extraction model, performing feature extraction on the plurality of enhanced images through the face feature extraction model to obtain a target feature image, comprising: Input the Nth enhanced image output by the Nth level face enhancement module and the N-1th enhanced image output by the N-1th level face enhancement module respectively through the Nth input terminal and the N-1th input terminal of the face feature extraction model, and splice them to obtain a spliced ​​feature image, input the spliced ​​feature image into the first level face feature extraction module, perform feature extraction on the Nth enhanced image through the first level face feature extraction module, and output a first feature image; Input the j-th enhanced image output by the j-th face enhancement module through the j-th input terminal of the face feature extraction model, and splice it with the j-1-th feature image output by the j-1-th face feature extraction module to obtain a spliced ​​feature image, input the spliced ​​feature image into the j-th face feature extraction module, perform feature extraction on the spliced ​​feature image through the j-th face feature extraction module, and output the j-th feature image; Inputting the N-1th feature image output by the N-1th face feature extraction module into the Nth face feature extraction module, performing feature extraction on the N-1th feature image through the Nth face feature extraction module, and outputting a target feature image; j is an integer greater than 1 and less than N.

4. The facial feature extraction method according to claim 2, characterized in that: The facial feature extraction module of each level from the first level to the N-1 level includes a cascaded residual unit and a downsampling unit, and the facial feature extraction module of the Nth level includes a residual unit; The residual unit is used to extract features from an image input into the unit; The residual unit submodule is used to perform down-sampling processing on the image input to the unit.

5. The method for extracting facial features according to any one of claims 1 to 4, characterized in that: The face enhancement model and face feature extraction model are trained in the following way: Acquire training sample data; the training sample data includes a plurality of groups of first sample images and second sample images with the same image content, and the image quality of the first sample image in each group of sample images is lower than the image quality of the second sample image; Based on the training sample data, the initial model combination and the initial discriminator are alternately and iteratively trained until the loss function value of the trained model combination and the loss function value of the discriminator both meet the preset convergence condition; The initial model combination includes: an initial face enhancement model and an initial face feature extraction model.

6. The method for extracting facial features according to claim 5, characterized in that: The initial model combination and the initial discriminator are alternately and iteratively trained based on the training sample data, including: In the stage of training the initial discriminator, for each group of sample images, the first sample image is input into the current face enhancement model, and the first sample image is subjected to face enhancement processing by the current face enhancement model to obtain a sample enhanced image; the current face enhancement model is the initial face enhancement model or the face enhancement model after the previous parameter adjustment; Inputting the sample enhanced image and the second sample image into a discriminator, and obtaining a first identification result of the sample enhanced image and an identification result of the second sample image through identification by the discriminator; Determining a first generative adversarial loss according to a first identification result of the sample enhanced image and an identification result of the second sample image; According to the first generative adversarial loss, the parameters of the initial discriminator are adjusted to obtain a discriminator after the parameters are adjusted.

7. The method for extracting facial features according to claim 6, characterized in that: Performing alternating iterative training on the initial model combination and the initial discriminator based on the training sample data, further comprising: In the stage of training the initial model combination, for each group of sample images, the sample enhanced image and the second sample image obtained based on the first sample image are respectively input into the current face feature extraction model, and the face features of the sample enhanced image and the second sample image are respectively extracted by the current face feature extraction model to obtain the extracted first sample feature vector and the second sample feature vector; the current face feature extraction model is the initial face feature extraction model or the face feature extraction model after the previous parameter adjustment; Inputting the sample enhanced image into the discriminator after the previous parameter adjustment, and obtaining a second identification result of the sample enhanced image through identification by the discriminator after the previous parameter adjustment; Determine a total image loss of the model combination according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image; The parameters of the current face enhancement model and the current face feature extraction model are adjusted according to the total image loss to obtain the face enhancement model and the face feature extraction model after the parameter adjustment.

8. The method for extracting facial features according to claim 7, characterized in that: The total image loss includes: feature loss, pixel loss and second generation adversarial loss; Determining the total image loss of the model combination according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image, comprising: Determining a feature loss of the first sample feature vector relative to the second sample feature vector; Determining a pixel loss of each pixel of the sample enhanced image relative to each pixel of the second sample image; A second generative adversarial loss is determined according to the second identification result.

9. The method for extracting facial features according to claim 8, characterized in that: The total image loss also includes: similarity loss; Determining the total image loss of the model combination according to the sample enhanced image, the first sample feature vector, the second sample feature vector, the second identification result, and the second sample image, further comprising: A similarity loss of the sample enhanced image relative to the second sample image is determined.

10. The method for extracting facial features according to claim 9, characterized in that: Determining a similarity loss of the sample enhanced image relative to the second sample image includes: Inputting the sample enhanced image and the second sample image into a key point detection module respectively, and obtaining a key point heat map of the sample enhanced image and a key point heat map of the second sample image output by the key point detection module; A similarity loss of the key point heat map of the sample enhanced image relative to the key point heat map of the sample image is determined as the similarity loss of the sample enhanced image relative to the sample image.

11. A facial feature extraction device using the facial feature extraction method according to any one of claims 1 to 10, characterized in that: include: An image acquisition module, used to acquire an image to be processed, wherein the image to be processed includes a face image; A first image processing module is used to input the image to be processed into the face enhancement model, and perform face enhancement processing on the image to be processed through the face enhancement model to obtain an enhanced image; the image quality of the enhanced image is higher than the image quality of the image to be processed; The second image processing module is used to input the enhanced image into the face feature extraction model, extract face features from the enhanced image through the face feature extraction model, and obtain a target feature image.

12. A facial feature extraction device, characterized in that: include: Memory; A processor, electrically connected to the memory; The memory stores a computer program, and the computer program is executed by the processor to implement the facial feature extraction method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the facial feature extraction method according to any one of claims 1 to 10 is implemented.

Images