Image face alignment method and device based on scene features, and electronic equipment

Abstract

The invention discloses an image face alignment method and device based on scene features and electronic equipment, and relates to the technical field of image processing.The method comprises the steps that a pre-trained editing vector task model is adopted to extract background features from a target scene corresponding to a shooting camera, the background features are converted into face attribute editing vectors, and the face attribute editing vectors are obtained; inputting the face image and the face attribute editing vector into a pre-trained image alignment model, outputting a standard angle face image corresponding to the face image, performing face feature matching on the shot face image when the image alignment model is trained, and performing matching on a target face image based on the face feature vector to obtain a target face image; and adjusting influence parameters when the standard angle face image is selected through the historical face features. According to the method and the device, the technical problems of lack of detail features and low image face recognition accuracy due to standard angle mapping performed by finding key points during face alignment in related technologies are solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology or other related fields, and in particular to a method and device for image face alignment based on scene features, and an electronic device. Background Art

[0002] Facial recognition technology has rapidly developed and been widely used in recent years. It has become an indispensable component in numerous fields, including security surveillance, mobile payments, and social media. Face alignment, a crucial preprocessing step for face recognition, has a crucial impact on the accuracy and efficiency of the overall system performance. However, current face alignment methods face numerous drawbacks, particularly regarding accuracy and stability in complex scenarios. For example, varying lighting conditions at different times of day and in different weather conditions can affect the quality of facial images, making it difficult to locate key feature points. Variations in the subject's head pose and expression also complicate face alignment, making traditional alignment methods prone to bias, particularly in non-frontal angles. Furthermore, existing face alignment algorithms rely on large amounts of annotated data to learn facial features. However, in fixed scenes, due to the stability of scene characteristics, the training data may lack diversity, impacting model robustness. Furthermore, while some techniques attempt to leverage scene-specific information to optimize face alignment, these techniques typically require training a separate model for each scene, which is time-consuming and labor-intensive, and difficult to scale widely. Furthermore, current face alignment algorithms rely on finding key points for standard angle mapping, which misses detailed features and requires continuous trajectories to utilize facial features from other angles.

[0003] Solutions in related technologies, such as traditional alignment methods based on manual features or advanced algorithms based on deep learning, often have limited effectiveness in facing the above challenges, especially when dealing with face alignment problems in fixed scenes, their performance may be significantly affected.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide a method and device for image face alignment based on scene features, and an electronic device, to at least solve the technical problems of finding key points for standard angle mapping, missing detailed features, and low accuracy of image face recognition when performing face alignment in related technologies.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a scene feature-based image face alignment method is provided, including: using a pre-trained editing vector task model to extract background features from the target scene corresponding to the shooting camera, and converting the background features into corresponding facial attribute editing vectors; inputting the captured target facial image and the facial attribute editing vector into a pre-trained image alignment model, and outputting a standard angle facial image corresponding to the target facial image, wherein the image alignment model includes a generator and a discriminator of an adversarial network, and when training the image alignment model, it includes an initial matching step and a historical facial image optimization step, the initial matching step includes performing facial feature matching on the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector, and the historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features.

[0007] Optionally, the initial matching step includes: using a fully convolutional network to perform feature matching on the captured historical facial image to generate a facial feature vector, wherein the angle difference between the current angle of the historical facial image and the frontal angle of the standard angle facial image is less than a preset angle difference threshold; inputting the facial feature vector into a fully connected layer to obtain network input parameters of the generator in the adversarial network, and combining them with a pre-input noise vector, the generator outputs a predicted standard angle facial image, wherein the network input parameters pass through multiple fully connected layers in the adversarial network generator to obtain a W space vector, and the W space vector is combined with the pre-input noise; inputting the predicted standard angle facial image into the discriminator of the adversarial network, and the discriminator performs downsampling processing and outputs a probability value in combination with the facial feature vector, wherein the probability value is used to characterize whether the predicted standard angle facial image is a successful matching image or a failed matching image corresponding to the historical facial image.

[0008] Optionally, the training method of the discriminator includes: receiving positive samples and negative samples, wherein the positive samples are real standard-angle face images and their corresponding face feature vectors, and the negative samples include two types: a combination of unmatched real face feature vectors and standard-angle face images, and an image generated by a generator based on a random noise vector and a processed face feature vector; through an alternating training strategy, updating the parameters of the discriminator based on the positive samples and negative samples, so that the discriminator distinguishes between real images and generated images, and distinguishes whether the face feature vector and the face image match; during the training process, the generator and the discriminator optimize each other through adversarial learning, the goal of the generator is to increase the recognition rate of the discriminator for generated images, and the goal of the discriminator is to reduce the recognition rate of generated images and improve the recognition rate of real images and matching feature vectors.

[0009] Optionally, the step of converting the background features into corresponding facial attribute editing vectors includes: extracting a background feature vector from an input scene image using a convolutional neural network, the background feature vector including at least one of the following: the geometric layout of the scene, lighting conditions, multiple character postures, and character positions; fusing the background feature vector with the facial feature vector, processing the background feature vector, the facial feature vector, and the vector difference through a fully connected layer to obtain a first vector, wherein the vector difference is the difference between the background feature vector and the facial feature vector, and the first vector is used to characterize the difference between the background feature vector and the facial feature vector; interactively processing the first vector and the facial feature vector using an attention mechanism to obtain a second vector, wherein the second vector is used to characterize the correlation between the background feature vector and the facial feature vector; inputting the second vector into a fully connected layer, limiting the range through a preset activation function, adding it to the first attribute editing vector, and then again through the preset activation function to generate a facial attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the background feature vector is input into the StyleFlow model.

[0010] Optionally, after generating the facial attribute editing vector, it also includes: calculating a loss function based on the facial attribute editing vector and a second attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after inputting the facial feature vector into the StyleFlow model; adjusting the parameters of the editing vector task model based on the loss function, and updating the editing vector task model.

[0011] Optionally, the historical face image optimization step includes: for the first training stage of the image alignment model, selecting pictures containing face videos from the input samples to obtain a historical image set, obtaining a common feature vector of all historical faces based on the historical image set, using the common feature vector to replace random noise, inputting it into the generator of the adversarial network, and the generator outputting a standard-angle face image corresponding to the historical face image; for the second training stage of the image alignment model, randomly selecting any image from the historical image set, inputting random noise and the selected any image into the generator of the adversarial network, and the generator outputting a standard-angle face image corresponding to the any image; during the first training stage and the second training stage of the image alignment model, alternately updating the parameters of the image alignment model and freezing some fully connected layers to adjust the dependence of the image alignment model on key facial features and the common feature vector; during the first training stage and the second training stage of the image alignment model, using a predetermined loss function to guide the training of the generator and discriminator in the adversarial network.

[0012] Optionally, the step of obtaining a common feature vector of all historical faces based on the historical image set includes: inputting the historical image set into a fully convolutional network, using the fully convolutional network to perform feature matching on the historical face images in the historical image set to generate a face feature vector; inputting the face feature vector into a fully connected layer to obtain key facial features, and splicing the key facial features, and inputting the spliced features into multiple convolutional layers to obtain a common feature vector of all historical faces.

[0013] According to another aspect of an embodiment of the present invention, a scene feature-based image face alignment device is also provided, including: a background feature conversion unit, used to use a pre-trained editing vector task model to extract background features from the target scene corresponding to the shooting camera, and convert the background features into corresponding facial attribute editing vectors; a face alignment unit, used to input the captured target facial image and the facial attribute editing vector into a pre-trained image alignment model, and output a standard angle facial image corresponding to the target facial image, wherein the image alignment model includes a generator and a discriminator of an adversarial network, and when training the image alignment model, it includes an initial matching step and a historical facial image optimization step, the initial matching step includes performing facial feature matching on the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector, and the historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features.

[0014] Optionally, when performing initial matching, the image face alignment device based on scene features includes: a feature matching unit, used to use a fully convolutional network to perform feature matching on the captured historical face image to generate a face feature vector, wherein the angle difference between the current angle of the historical face image and the frontal angle of the standard angle face image is less than a preset angle difference threshold; a first feature vector input unit, used to input the face feature vector into a fully connected layer to obtain network input parameters of the generator in the adversarial network, and combine the pre-input noise vector to output a predicted standard angle face image by the generator, wherein the network input parameters pass through multiple fully connected layers in the adversarial network generator to obtain a W space vector, and the W space vector is combined with the pre-input noise; a first face image input unit, used to input the predicted standard angle face image into the discriminator of the adversarial network, and after the discriminator performs downsampling processing, it combines the face feature vector to output a probability value, wherein the probability value is used to characterize whether the predicted standard angle face image is a successful matching image or a failed matching image corresponding to the historical face image.

[0015] Optionally, when training the discriminator, the image face alignment device based on scene features includes: a sample receiving unit for receiving positive samples and negative samples, wherein the positive samples are real standard-angle face images and their corresponding face feature vectors, and the negative samples include two types: a combination of unmatched real face feature vectors and standard-angle face images, and an image generated by a generator based on a random noise vector and a processed face feature vector; a discriminator training unit for updating the parameters of the discriminator based on the positive samples and negative samples through an alternating training strategy, so that the discriminator distinguishes between real images and generated images, and distinguishes whether the face feature vector and the face image match; an optimization unit for mutually optimizing the generator and the discriminator through adversarial learning during the training process, wherein the goal of the generator is to increase the recognition rate of the discriminator for the generated images, and the goal of the discriminator is to reduce the recognition rate of the generated images and improve the recognition rate of the real images and matching feature vectors.

[0016] Optionally, the background feature conversion unit includes: a background feature vector extraction module for extracting a background feature vector from an input scene image using a convolutional neural network, wherein the background feature vector includes at least one of the following: the geometric layout of the scene, lighting conditions, multiple character postures, and character positions; a feature vector fusion module for fusing the background feature vector with the face feature vector, processing the background feature vector, the face feature vector, and the vector difference through a fully connected layer to obtain a first vector, wherein the vector difference is the difference between the background feature vector and the face feature vector, and the first vector is used to represent the difference between the background feature vector and the face feature vector. The difference between the quantities; a vector interaction module, used to use the attention mechanism to interactively process the first vector and the facial feature vector to obtain a second vector, wherein the second vector is used to characterize the correlation between the background feature vector and the facial feature vector; a vector input module, used to input the second vector into the fully connected layer, limit the range through a preset activation function, and add it to the first attribute editing vector, and then generate a facial attribute editing vector through the preset activation function again, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the background feature vector is input into the StyleFlow model.

[0017] Optionally, the scene feature-based image face alignment device also includes: a loss function calculation unit, which is used to calculate the loss function based on the face attribute editing vector and the second attribute editing vector after generating the face attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after inputting the face feature vector into the StyleFlow model; a model updating unit, which is used to adjust the parameters of the editing vector task model based on the loss function and update the editing vector task model.

[0018] Optionally, when optimizing historical face images, the image face alignment device based on scene features includes: a first picture selection unit for selecting pictures containing face videos from the input samples for the first training phase of the image alignment model to obtain a historical image set, obtaining a common feature vector of all historical faces based on the historical image set, using the common feature vector to replace random noise, and inputting it into a generator of an adversarial network, which outputs a standard angle face image corresponding to the historical face image; a second picture selection unit for randomly selecting any picture from the historical image set for the second training phase of the image alignment model. An image, random noise and any selected image are input into the generator of the adversarial network, and the generator outputs a standard angle face image corresponding to any image; a model parameter updating unit is used to alternately update the parameters of the image alignment model and freeze some fully connected layers during the first training stage and the second training stage of the image alignment model, and adjust the dependence of the image alignment model on the key features of the face and the common feature vector; a loss function guidance unit is used to use a predetermined loss function to guide the training of the generator and discriminator in the adversarial network during the first training stage and the second training stage of the image alignment model.

[0019] Optionally, the first image selection unit includes: an image set input module, used to input the historical image set into a fully convolutional network, use the fully convolutional network to perform feature matching on the historical facial images in the historical image set, and generate a facial feature vector; a key feature splicing module, used to input the facial feature vector into a fully connected layer to obtain facial key features, and splice the facial key features, and input the spliced features into multiple convolutional layers to obtain a common feature vector for all historical faces.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned scene feature-based image face alignment methods.

[0021] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned scene feature-based image face alignment methods.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is further provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the above-mentioned methods for image face alignment based on scene features are implemented.

[0023] In the present disclosure, a pre-trained editing vector task model is used to extract background features from the target scene corresponding to the shooting camera, and the background features are converted into corresponding facial attribute editing vectors. The captured target facial image and the facial attribute editing vector are input into a pre-trained image alignment model, and a standard angle facial image corresponding to the target facial image is output. The image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical facial image optimization step. The initial matching step includes performing facial feature matching on the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector. The historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features.

[0024] Based on the above-mentioned public content, the background features obtained by the camera are converted into facial attribute editing vectors, and the detailed attribute features of the face can be obtained to assist in face prediction. At the same time, the image alignment model is optimized through the facial historical feature image, thereby improving the accuracy of the model in matching standard angle facial images, thereby solving the technical problems of finding key points for standard angle mapping, missing detailed features, and image face recognition accuracy when performing face alignment in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing an image face alignment method based on scene features is shown;

[0027] Figure 2 is a flowchart of an optional method for image face alignment based on scene features according to an embodiment of the present invention;

[0028] Figure 3 This is an optional standard angle face generation task model structure diagram according to an embodiment of the present invention;

[0029] Figure 4 is a structural diagram of an optional background feature generation editing vector task model according to an embodiment of the present invention;

[0030] Figure 5 1 is a structural diagram of an optional task model for the influence of historical facial features on generating standard-angle faces according to an embodiment of the present invention;

[0031] Figure 6 is a schematic diagram of an optional image face alignment device based on scene features according to an embodiment of the present invention;

[0032] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] To facilitate those skilled in the art to understand the present invention, some of the terms or nouns involved in the embodiments of the present invention are explained below:

[0036] The StyleFlow model, an image attribute editing method based on a pre-trained generative network StyleGAN, allows changing specific attributes in an image, such as facial posture, age, and expression, by simply pushing and pulling vectors, without directly manipulating pixels or manually annotating data.

[0037] A fully connected layer, or FC, is a basic component in a neural network. Every neuron in each layer is connected to all neurons in the previous layer. During the calculation process, the fully connected layer can be regarded as a linear transformation of a multidimensional vector. Through matrix multiplication and bias addition, the output of the previous layer is converted into the output of this layer. In the present invention, the fully connected layer is used to process and transform the extracted facial features to generate edit vectors related to facial attributes. These fully connected layers may contain multiple neurons, each with its own weights and biases. By training these parameters, the model can better understand and utilize facial features, thereby improving the effect of face alignment.

[0038] Downsampling, in the field of image processing and computer vision, usually refers to the process of reducing the resolution of an image or reducing the data size, which is called "downsampling" or "downsampling".

[0039] The goal of the generator is to create images that look like real faces. Its input is usually a feature vector processed by a CNN (convolutional neural network) and an additional noise vector. The output of the generator is a predicted image, which is a normalized face image generated based on the input facial feature vector and noise.

[0040] The discriminator distinguishes between images generated by the generator and real images. It receives an image as input, whether it is generated by the generator or taken from a real dataset. The discriminator outputs a probability value, indicating the probability that the input image is a real image rather than a fake image generated by the generator.

[0041] The predicted standard angle image, pred pic, is the output of the generator. In tasks involving image generation or image-to-image translation, such as the case of StyleGAN for face generation, the generator's task is to generate a new image based on the input feature vector or noise.

[0042] It should be noted that the scene feature-based image face alignment method and device thereof in the present disclosure can be used in the field of image processing technology. When face alignment is achieved based on camera fixed scene features and historical face features, it can also be used in any field other than the field of image processing technology. When face alignment is achieved based on camera fixed scene features and historical face features, the present disclosure does not limit the application field of the scene feature-based image face alignment method and device thereof.

[0043] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and the relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0044] It should be noted that in this disclosure, when collecting and analyzing customer information, the corresponding operation entrance is provided for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0045] The following embodiments of the present invention can be applied to various systems, applications, and devices for scene-based image face alignment. This invention optimizes face alignment by leveraging historical facial features and fixed scene features, providing more accurate features for face comparison. It can also restore faces at standard angles for other purposes, significantly improving face comparison accuracy, increasing intrusion detection rates, and reducing false detection rates, ultimately optimizing intelligent camera products.

[0046] The present invention can be applied to intelligent monitoring and facial recognition systems in various fixed scenarios, where the camera position and angle are relatively stable. This allows for the utilization of fixed scene features and historical facial features captured by the camera. Examples include security monitoring systems, access control systems, and mobile terminal identity recognition. For example, in public security monitoring systems, fixed cameras are often used for real-time monitoring and access management. The environments in which these cameras are located (e.g., corridors, doorways, and elevators) have stable background features, such as wall color, sign shape, and light direction. By applying the present invention, the camera can leverage these fixed scene features and historical facial data to optimize the face detection and recognition process in real time, significantly improving recognition accuracy and efficiency and effectively reducing false positives and false negatives. For example, the system can more accurately identify employees who frequently enter a building, even if they appear under varying lighting conditions or are slightly obscured.

[0047] The technical solution provided by this invention significantly improves facial recognition accuracy by optimizing a face alignment algorithm using fixed scene features and historical facial data. No longer limited to the facial features of a single image, the system can now perform more accurate comparisons based on the combined context of the scene and historical facial data, reducing facial recognition errors caused by non-ideal conditions such as posture, expression, and lighting. The optimized face alignment algorithm helps reduce false positives and false negatives in facial recognition systems. This can significantly improve system reliability and security, particularly in security monitoring and access control scenarios, reducing unnecessary alerts and access delays.

[0048] In addition, the face alignment algorithm provided by the present invention enhances the robustness and adaptability of the face recognition system in complex environments. Whether it is changeable outdoor weather or dynamic indoor lighting, the system can effectively cope with it and maintain stable performance.

[0049] The present invention will be described in detail below with reference to various embodiments.

[0050] Example 1

[0051] According to an embodiment of the present invention, an embodiment of a method for image face alignment based on scene features is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] The embodiment of the image face alignment method based on scene features provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing an image face alignment method based on scene features is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1The computer system includes a processor 102 (illustrated as 102a, 102b, ..., 102n) (the processor 102 may include, but is not limited to, a microcontroller unit (MCU) or a field programmable gate array (FPGA)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, the computer system may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS), a network interface, a power supply, and / or a camera. Those skilled in the art will appreciate that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0053] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." This data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be fully or partially integrated into any of the other components of the computer terminal 10 (or mobile device). As discussed in the embodiments of this application, this data processing circuitry functions as a processor control (e.g., selecting a variable resistor terminal path connected to an interface).

[0054] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the scene feature-based image face alignment method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the above-mentioned scene feature-based image face alignment method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] Transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of computer terminal 10. In one embodiment, transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0056] The display may be, for example, a touch screen liquid crystal display (LCD), which enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0057] Under the above operating environment, this application provides Figure 2 The image face alignment method based on scene features is shown. Figure 2 FIG. 1 is a flow chart of an optional method for image face alignment based on scene features according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0058] First, the embodiment of the present invention uses a pre-trained CNN model (convolutional neural network) to extract background features from the target scene image.

[0059] Step S201 : Using a pre-trained edit vector task model, background features are extracted from a target scene corresponding to a shooting camera, and the background features are converted into corresponding facial attribute edit vectors.

[0060] The scene background features mentioned in this embodiment include, but are not limited to, the scene's geometric layout, lighting conditions, and common poses and positional information of characters. Optionally, in the process of extracting background features, the image can be subjected to convolution, pooling, and other operations using a multi-layer CNN structure to gradually extract high-level features of the image, thereby obtaining a comprehensive description of the scene background. Next, the background features need to be converted into corresponding facial attribute edit vectors. Specifically, a pre-trained edit vector task model (e.g., a StyleFlow model) can be used to obtain the facial attribute edit vector corresponding to the current image, i.e., the facial edit vector corresponding to the current background features.

[0061] In an alternative example, consider a camera capturing a surveillance image of a company entrance. The CNN model extracts specific background features from the image, such as the entrance's lighting conditions and employees' typical walking paths. These features are converted into edit vectors, which guide the image alignment model to account for this context when generating standard-angle face images, thereby improving alignment accuracy.

[0062] Step S202: input the captured target face image and the face attribute editing vector into a pre-trained image alignment model, and output a standard angle face image corresponding to the target face image, wherein the image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical face image optimization step. The initial matching step includes performing facial feature matching on the captured target face image to obtain a facial feature vector, and matching the target face image based on the facial feature vector. The historical face image optimization step includes adjusting the influencing parameters when selecting the standard angle face image through historical facial features.

[0063] After inputting the target face image, the image alignment model first matches facial features on the image using pre-trained face detection and feature extraction modules to obtain a facial feature vector. This process may include key point location and facial contour detection. Based on the facial feature vector, the image alignment model's generator initially generates a standard-angle face image. The generator uses the facial feature vector as input to generate a preliminary standard-angle face image, with the generator's weights based on the training results of Task 1. The model then uses edit vectors generated from historical facial feature vectors and background features to further optimize the generated standard-angle face image by training the generator on the influence of historical facial features.

[0064] It should be noted that when training the image alignment model, the generator and discriminator of the adversarial network are used. The generator is responsible for generating face images with standard angles, while the discriminator is responsible for evaluating the authenticity and quality of the generated images.

[0065] In one example, a camera captures an employee entering an office. First, the model generates a preliminary standard-angle facial image based on the employee's facial feature vector. Then, using the edit vector generated from the office's long-accumulated historical facial features and current background features, the model optimizes the preliminary image to produce a more accurate and natural standard-angle facial image. This allows the image alignment model of the present invention to be applied to perform face detection and feature extraction, matching facial features across images, improving image alignment quality and reducing false positives due to factors such as posture, expression, and lighting.

[0066] Through the above steps, a pre-trained editing vector task model can be used to extract background features from the target scene corresponding to the shooting camera, and the background features can be converted into corresponding facial attribute editing vectors. The captured target facial image and the facial attribute editing vector are input into the pre-trained image alignment model, and a standard angle facial image corresponding to the target facial image is output. The image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical facial image optimization step. The initial matching step includes facial feature matching of the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector. The historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features. In this embodiment, the background features obtained by the camera can be converted into facial attribute editing vectors, and facial detail attribute features can be obtained to assist in face prediction. At the same time, the image alignment model is optimized through the facial historical feature image, thereby improving the accuracy of the model in matching standard angle facial images, thereby solving the technical problems of finding key points for standard angle mapping, missing detail features, and image face recognition accuracy when performing face alignment in related technologies.

[0067] Optionally, the initial matching step includes: using a fully convolutional network to perform feature matching on the captured historical facial image to generate a facial feature vector, wherein the angle difference between the current angle of the historical facial image and the frontal angle of the standard angle facial image is less than a preset angle difference threshold; inputting the facial feature vector into the fully connected layer to obtain the network input parameters of the generator in the adversarial network, and combining them with the pre-input noise vector, the generator outputs a predicted standard angle facial image, wherein the network input parameters pass through multiple fully connected layers in the adversarial network generator to obtain a W space vector, and the W space vector is combined with the pre-input noise; inputting the predicted standard angle facial image into the discriminator of the adversarial network, and the discriminator performs downsampling processing and outputs a probability value in combination with the facial feature vector, wherein the probability value is used to characterize whether the predicted standard angle facial image is a successful matching image or a failed matching image corresponding to the historical facial image.

[0068] When matching historical facial image features, a fully convolutional network (FCN) is first used to extract features from the historical facial images captured by the camera. This network can extract both global and local features from images without the need for fixed-size image cropping. In this embodiment, the FCN is used to process facial images of varying angles and sizes to generate efficient and accurate facial feature vectors.

[0069] Furthermore, the angle information of the historical facial image is compared with the standard angle (frontal angle). If the angle difference between the current angle of the historical facial image and the frontal angle of the standard angle facial image is less than a preset angle difference threshold, this indicates that the image is suitable for preliminary matching for face alignment. The preset angle difference threshold can be adjusted based on the specific application scenario and the required face recognition accuracy to ensure that the selected image provides sufficient facial information to support subsequent alignment operations.

[0070] The generated facial feature vector (e) is input to a fully connected layer (FC layer). After a series of neural network operations, it is converted into the network input parameters of the generator in the adversarial network. These parameters are typically represented in a vector space that captures key features and attributes of the face, such as facial contours, eye position, and nose shape, providing the generator with the necessary information to generate frontal facial images. Within the generator, these network input parameters are further processed through multiple fully connected layers, transforming them into W-space vectors. In StyleGAN and similar GAN architectures, W-space is a more efficient intermediate representation that separates the style and content of generated images, making facial image editing (such as adjusting angle or pose) more precise and controllable. In the generator, the W-space vector is combined with a pre-input noise vector. The generator leverages this information and previously learned patterns to output a predicted face image from a standard angle. The introduction of the noise vector increases the diversity of the generated images, prevents overly stereotyped results, and makes the generated facial images more realistic and natural.

[0071] The predicted standard-angle face image is input into the adversarial network's discriminator, which downsamples the image to reduce computational effort while preserving key image features. After downsampling, the discriminator evaluates the image based on the facial feature vector and outputs a probability value. This probability value determines whether the predicted image successfully matches the historical face image—that is, whether the generated image is a realistic, frontal-angle face image. The discriminator's output probability value is calculated based on the image's authenticity and the degree of match between the facial feature vector and the historical face image. If the generated image closely matches the historical face image and appears realistic and natural, the discriminator's output probability value will be close to 1. Conversely, if the generated image is of low quality or does not match the historical face image, the probability value will be close to 0. This probability value guides the training and optimization of the generator, ensuring that the generated image is both realistic and accurately reflects the frontal-angle features of the historical face image.

[0072] Through the above process, the initial matching step provides a preliminary standard-angle face image for the subsequent historical face image optimization step. The quality and matching degree of this preliminary image depend on the precise design and coordinated operation of the fully convolutional network, fully connected layers, generator, and discriminator. When using historical face image features for preliminary matching, the embodiments of the present invention effectively utilize prior knowledge of fixed scenes and the rich information of historical face data to improve the efficiency and accuracy of face alignment and reduce the false detection rate, meeting the high requirements of face recognition systems in practical applications and the robustness requirements in complex environments.

[0073] Optionally, the training method of the discriminator includes: receiving positive samples and negative samples, wherein the positive samples are real standard-angle face images and their corresponding face feature vectors, and the negative samples include two types: a combination of unmatched real face feature vectors and standard-angle face images, and an image generated by a generator based on a random noise vector and a processed face feature vector; through an alternating training strategy, updating the parameters of the discriminator based on the positive samples and the negative samples, so that the discriminator distinguishes between real images and generated images, and distinguishes whether the face feature vector matches the face image; during the training process, the generator and the discriminator optimize each other through adversarial learning, the goal of the generator is to increase the recognition rate of the discriminator for generated images, and the goal of the discriminator is to reduce the recognition rate of generated images and improve the recognition rate of real images and matching feature vectors.

[0074] The discriminator receives two types of samples as input: 1. Positive samples: These samples consist of real-world facial images from standard angles and their corresponding facial feature vectors. These samples serve as a benchmark for training the discriminator to identify real images and correctly match facial features. 2. Negative samples: These are further subdivided into two types: combinations of mismatched real-world facial feature vectors and standard-angle facial images. These samples are used to verify the discriminator's ability to detect mismatches between feature vectors and images, preventing the generator from outputting facial attributes that differ from the real image.

[0075] It should be noted that an alternating training strategy can be employed during the training of the discriminator. The training process is as follows: First, the discriminator updates its parameters based on positive samples and a first type of negative samples (combinations of mismatched facial feature vectors and standard-angle face images). This allows the discriminator to learn to distinguish between real face images and correctly matched feature vector combinations, while also identifying mismatched feature vector and image combinations, ensuring the model's ability to accurately discriminate facial attributes. Next, the discriminator is trained again using images generated by the generator as a second type of negative samples, enhancing its ability to distinguish generated images. The discriminator's goal is to learn to distinguish between images forged by the generator and real ones, thereby reducing the recognition rate of images output by the generator. During training, the generator and discriminator engage in continuous adversarial learning. The generator's goal is to increase the discriminator's recognition rate for generated images. By adjusting its internal parameters, the generator gradually improves the fidelity of its generated images and the accuracy of facial features, until the discriminator cannot easily distinguish between generated and real images. The goal of the discriminator is to reduce the recognition rate of generated images and improve the recognition rate of real images and matching feature vectors. The discriminator continuously improves its ability to distinguish real from fake images and feature matching combinations by learning the patterns of the generator.

[0076] Through adversarial training strategies and loss functions, the model learns how to generate high-quality, standard-angle facial images that conform to facial attributes, while effectively distinguishing between real and fake images and feature matching, thereby optimizing the face alignment algorithm. This optimization not only improves the image alignment quality but also enhances the model's robustness and adaptability in complex environments.

[0077] Optionally, the step of converting background features into corresponding facial attribute editing vectors includes: using a convolutional neural network to extract a background feature vector from an input scene image, the background feature vector including at least one of the following: the geometric layout of the scene, lighting conditions, multiple character postures, and character positions; fusing the background feature vector with the facial feature vector, processing the background feature vector, the facial feature vector, and the vector difference through a fully connected layer to obtain a first vector, wherein the vector difference is the difference between the background feature vector and the facial feature vector, and the first vector is used to characterize the difference between the background feature vector and the facial feature vector; using an attention mechanism to interactively process the first vector and the facial feature vector to obtain a second vector, wherein the second vector is used to characterize the correlation between the background feature vector and the facial feature vector; inputting the second vector into a fully connected layer, limiting the range through a preset activation function, and adding it to the first attribute editing vector, and then again through the preset activation function to generate a facial attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the background feature vector is input into the StyleFlow model.

[0078] In embodiments of the present invention, deep learning techniques (such as convolutional neural networks (CNNs)) combined with attention mechanisms can be used to transform scene geometry, lighting conditions, various character poses, and positions into attribute edit vectors that can guide face alignment. To extract background feature vectors, a CNN can be used from the input scene image. Through multiple layers of convolution and pooling, the CNN can identify and extract important features within the image. When processing fixed-camera scenes, the CNN can learn the scene geometry (e.g., the relative positions of walls, doors, and windows), lighting conditions (e.g., the direction and intensity of the light source), common character poses (standing, walking, running, etc.), and the character's position within the scene. The extracted background feature vector is then fused with the facial feature vector. First, a difference vector is calculated between the background feature vector and the facial feature vector to generate a first vector. This vector represents the difference between the background feature vector and the facial feature vector, specifically how the scene background affects the character's appearance, such as the effect of lighting on facial shadows and the potential constraints imposed by the scene layout on the character's pose. Then, is input into the fully connected layer together with the facial feature vector, and the fully connected layer processes them and the difference between them to obtain a more comprehensive first vector, which further strengthens the analysis of the impact of background features on facial attributes.

[0079] In addition, an embodiment of the present invention also introduces an attention mechanism to interactively process the first vector and the facial feature vector to obtain a second vector. The role of the attention mechanism here is that it can identify and emphasize those background feature parts that are most relevant to facial attribute editing. For example, if there is strong side light in the scene, the attention mechanism will assign more weight to the lighting condition feature, thereby ensuring that the generated facial attribute editing vector can accurately reflect the impact of lighting changes on facial appearance. Through such interactive processing, the second vector can more accurately characterize the correlation between the background feature vector and the facial feature vector, that is, how the background affects or should affect the editing of facial attributes.

[0080] The second vector is added to the first attribute editing vector obtained from the attribute classifier of the StyleFlow model, and the final face attribute editing vector (preds) is generated again through a preset activation function. This editing vector incorporates the influence of background features, so that when performing face alignment, the model can dynamically adjust the alignment strategy according to the specific scene conditions, thereby generating a face image that is closer to the standard angle of the real scene, thereby improving the accuracy and robustness of face recognition.

[0081] Imagine a camera capturing an employee entering an office. The scene might include elements such as desks, windows, and lamps. Furthermore, since it's afternoon, natural light enters through a side window, creating some shadows. Through the aforementioned steps, a CNN and attention mechanism automatically identify these background features and generate an edit vector that not only incorporates information about the lighting conditions but also takes into account the person's position and pose in the scene. When the model generates a standard-angle facial image based on a profile image of the employee, the edit vector guides the generator in handling lighting and shadow effects, ensuring that the generated facial image matches the actual scene while preserving the person's pose and position, ultimately achieving more accurate face alignment.

[0082] Optionally, after generating the facial attribute editing vector, it also includes: calculating a loss function based on the facial attribute editing vector and a second attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the facial feature vector is input into the StyleFlow model; adjusting the parameters of the editing vector task model based on the loss function, and updating the editing vector task model.

[0083] After generating the facial attribute edit vector, the concept of a second attribute edit vector is introduced to enhance the model training and optimization process. The second attribute edit vector, also known as the first attribute edit vector, is obtained by inputting the facial feature vector (e) into the StyleFlow model and then using its image attribute classification module. The StyleFlow model understands the inherent variations in facial attributes, such as head pose, lighting, and expression, while the image attribute classification module is responsible for predicting the value or category of these attributes.

[0084] Optionally, the historical face image optimization step includes: for the first training stage of the image alignment model, selecting pictures containing face videos in the input samples to obtain a historical image set, obtaining a common feature vector of all historical faces based on the historical image set, using the common feature vector to replace random noise, inputting it into the generator of the adversarial network, and the generator outputting a standard angle face image corresponding to the historical face image; for the second training stage of the image alignment model, randomly selecting any image in the historical image set, inputting random noise and any selected image into the generator of the adversarial network, and the generator outputting a standard angle face image corresponding to any image; during the first training stage and the second training stage of the image alignment model, alternately updating the parameters of the image alignment model and freezing some fully connected layers to adjust the image alignment model's dependence on key facial features and common feature vectors; during the first training stage and the second training stage of the image alignment model, using a predetermined loss function to guide the training of the generator and discriminator in the adversarial network.

[0085] The first training phase of the image alignment model focuses on processing a historical image collection. Input samples are selected from images containing facial videos to generate a historical image collection. By using CNNs and fully connected layers to extract and fuse features from the historical image collection, a common feature vector for all historical faces is obtained. The second training phase focuses more on the model's ability to process a single historical image. In this phase, an image is randomly selected from the historical image collection and fed into the adversarial network generator along with random noise (z) and a facial feature vector (e). The generator's task is to output a standard-angle facial image corresponding to the randomly selected image, taking into account the background features and historical information of the randomly selected image. This phase of training helps the model better understand and process individual differences and how to generate more natural and accurate facial images under specific background and historical conditions.

[0086] During the first and second training phases, an alternating update and freeze strategy can be used to adjust the image alignment model's reliance on key facial features and common eigenvectors. Specifically, the image alignment model's parameters can be updated in some training rounds to improve model performance, while in other rounds, the parameters of some fully connected layers are frozen, allowing the model to focus on learning how to optimize under the influence of different historical images and background features. This helps the model flexibly respond to face alignment needs in different scenarios while preserving facial identity information.

[0087] Optionally, the step of obtaining a common feature vector of all historical faces based on a historical image set includes: inputting the historical image set into a fully convolutional network, using the fully convolutional network to perform feature matching on historical face images in the historical image set to generate a face feature vector; inputting the face feature vector into a fully connected layer to obtain key facial features, and splicing the key facial features, and inputting the spliced features into multiple convolutional layers to obtain a common feature vector of all historical faces.

[0088] A fully convolutional network (FCN) is a neural network structure that can process input images of any size and output feature maps of the same size. In this step, the FCN is used to extract features from a historical image collection pixel by pixel, generating a series of facial feature vectors. Compared to traditional convolutional neural networks, the FCN can retain more detailed features without compromising image size, which is crucial for locating facial landmarks and capturing subtle expressions and postures. Through multiple layers of convolution and pooling operations, the FCN gradually converts the image into a high-level feature representation. These feature vectors encompass multiple aspects of the face, including geometric structure, texture details, lighting effects, and possible variations in posture and expression. By matching facial features in historical images, the FCN can capture the general patterns of facial features in this specific scenario.

[0089] For each historical facial image, the FCN ultimately outputs a feature vector that summarizes all key information about the image. The multiple feature vectors output by the FCN are fed into fully connected layers, which further refine and fuse these features to comprehensively assess facial attributes from multiple dimensions, including but not limited to facial contours, the relative positions of key points, skin tone, and nuances of expression. After processing by the FC layers, the resulting features are called key facial features, which provide a high-level summary of facial attributes. Since a historical image collection may contain multiple facial images, each corresponding to a feature vector, these feature vectors need to be concatenated to create a comprehensive vector encompassing all historical facial features. This concatenation is achieved by concatenating different feature vectors to form a multi-dimensional vector space, where each dimension reflects a specific aspect of the historical face. The concatenated feature vectors are then fed into a series of convolutional layers. These layers are designed to perform further deep feature learning and fusion on the concatenated feature vectors. Through local connections and weight sharing, the convolutional layers efficiently extract and process spatial-level features of the image, thereby enhancing the model's generalization capabilities.

[0090] In practice, the design of fully convolutional networks and fully connected layers must take into account the unique characteristics of facial features. For example, the network depth and width, the choice of activation function, and the regularization strategy all require careful tuning. The parameter settings of the convolutional layers should also fully consider the diversity of historical data to ensure that the model can learn universal facial feature representations from facial images of different angles, expressions, and lighting conditions.

[0091] Furthermore, the generation of common feature vectors depends not only on the volume of historical image data but also on the quality and diversity of the images. Therefore, in practice, it is necessary to collect a sufficient amount of high-quality historical facial data to ensure the representativeness and accuracy of the common feature vectors. For real-time applications, the common feature vectors can be updated and adjusted in real time, allowing them to continuously adapt to subtle changes in the environment, enhancing the system's real-time performance and flexibility.

[0092] The following describes in detail another optional specific implementation.

[0093] This implementation scheme first needs to build a model to generate a standard angle for faces at any angle, then add the influence of background features on its generation based on this model, and finally design a structure that can use historical face features to optimize the current generation results.

[0094] The solution is divided into three tasks. The first part is the task of generating standard angle faces from any angle, and the main structure can be based on styleGAN; the second part is the task of generating edit vectors from background features, and the main structure is based on StyleFlow; the third part is the task of training the influence of historical facial features on the generation of standard angle faces, and the main structure can also be based on StyleGAN.

[0095] Task 1: Standard angle face generation task.

[0096] The goal of this task is to generate an image of a face at a standard angle from a face at any angle. Fully convolutional layers are used to generate facial features, which are then passed through a fully connected layer as the StyleGAN input Z (the input to the GAN network). The discriminator inputs positive samples: real images and their corresponding facial features, while negative samples include two types: a matching standard-angle face image with mismatched facial features, and a generated image with matching facial features.

[0097] Figure 3 is an optional standard angle face generation task model structure diagram according to an embodiment of the present invention, such as Figure 3As shown in the figure, after the face picture is input into the CNN (fully convolutional network), the background features are extracted and converted into a face feature vector (Embedding), which is then input into the fully connected layer (FC). After that, it is sent to the generator in the adversarial network. With Z as the input, it passes through 8 fully connected layers FC to obtain the w space. Then, after passing through N network layers and combining with the input noise, the predicted standard angle face image is output. The image is sent to the discriminator of the adversarial network. After downsampling, it is combined with the face feature vector (abbreviated as e) to output the discrimination probability value. The discrimination probability value is used to confirm whether the predicted standard angle face image is the true standard angle face image corresponding to the input face picture.

[0098] The loss calculation formula of the model is as follows:

[0099] Loss_G=log(exp(-D(G(z),e))+1)

[0100] Loss_D=log(exp(D(G(z),e))+1)+log(exp(-D(x, e))+1).

[0101] Where Loss_G represents the loss of the generator and Loss_D represents the loss of the discriminator.

[0102] Task 2: Background feature generation editing vector task.

[0103] The second task aims to convert background features into corresponding facial attribute edit vectors. This requires using the trained styleFlow model to obtain the facial attribute edit vectors corresponding to the current image, specifically the facial edit vectors corresponding to the current background features. This example provides a model structure that focuses on the impact of background features on the conversion of current facial features into corresponding edit vectors.

[0104] Figure 4 is a structural diagram of an optional background feature generation editing vector task model according to an embodiment of the present invention, such as Figure 4 As shown, the background and face are passed through the same CNN model to obtain vectors embedding1 and embedding2. The input of styleFlow is replaced with embedding1 and embedding2. The attribute edit vectors obtained by the attribute classifier of styleFlow are recorded as label1 and label2. Embedding1, embedding2 and their difference are spliced together and passed through the fully connected layer to obtain h0, which represents the difference between background features and face features:

[0105] h0=FC(concat[embedding1, embedding2, embedding2- embedding1]).

[0106] The generated h0 and embedding2 are passed through attention to get h1, which represents the difference between background features and facial features and the relationship between facial features. The input of attention is the query vector Q, key vector K, and value vector V. Here h0 is used as input Q, and embedding2 is used as K and V:

[0107] h1=attention(h0, embedding2, embedding2).

[0108] The generated h1 is passed through the fully connected layer to obtain a vector of the same dimension as label1. The size is limited to between -1 and 1 by the activation function tanh. The result is added to label1 and then passed through tanh to obtain the final output preds, which is then calculated with label2. The formula is as follows:

[0109] loss=log(cosh(preds-label2)).

[0110] This task model and the original styleFlow model should be frozen and trained alternately.

[0111] Task 3: The task of studying the influence of historical facial features on the generation of standard angle faces.

[0112] Here, Task 3 requires the use of the model parameters that have been trained in Task 1. Each input sample is divided into historical face images (positive samples of the same person and negative samples of different people) and the current face image.

[0113] Figure 5 FIG. 1 is a structural diagram of a task model for determining the influence of facial history features on generating standard-angle faces according to an embodiment of the present invention. Figure 5As shown in the figure, the task model generation process is divided into two parts. The first part selects pic1-picK samples from the input sample (such as a continuous video image containing a face). Based on task one, the generator passes the facial feature vector e through eight fully connected layers to obtain high-level semantic information (expressing the key features of the model). It is then concatenated and passed through four convolutional layers to obtain the common features of all historical faces. The resulting vector is then fed into the network instead of noise. Here, the pic1-picK samples all use the same noise (the noise in the styleGAN network increases randomness). The ultimate goal is to enable the generator to use this noise to generate images with corresponding labels (constructing historical feature images corresponding to the randomness represented by the noise).

[0114] The second part uses the noise generated in the first part to randomly select picN and enter the network. The final goal is to let the generator use this noise to generate an image with the corresponding label when picN is input.

[0115] During the model training process, the parameters of the virtual box are not updated. The loss calculation formulas for the two parts are the same, as follows:

[0116] Loss_G=log(exp(-D(G(z)))+1)

[0117] Loss_D=log(exp(D(G(z)))+1)+log(exp(-D(x))+1).

[0118] Through the above embodiments, the optimization of the face alignment algorithm provided by the present invention enhances the robustness and adaptability of the face recognition system in complex environments. Whether it is changeable outdoor weather or dynamic indoor lighting, the system can effectively cope with it and maintain stable performance.

[0119] The following describes it in detail with reference to another embodiment.

[0120] Example 2

[0121] The image face alignment device based on scene features provided in this embodiment includes multiple implementation units, each implementation unit corresponds to each implementation step in the above-mentioned embodiment 1. Its specific implementation method and beneficial effects can be referred to the above-mentioned method embodiment and will not be repeated here.

[0122] Figure 6 FIG. 1 is a schematic diagram of an optional image face alignment device based on scene features according to an embodiment of the present invention. Figure 6 As shown, the image face alignment device based on scene features may include: a background feature conversion unit 61 and a face alignment unit 62.

[0123] The background feature conversion unit 61 is used to extract background features from the target scene corresponding to the shooting camera using a pre-trained editing vector task model, and convert the background features into corresponding facial attribute editing vectors.

[0124] The face alignment unit 62 is used to input the captured target face image and the face attribute editing vector into a pre-trained image alignment model, and output a standard angle face image corresponding to the target face image, wherein the image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical face image optimization step. The initial matching step includes performing facial feature matching on the captured target face image to obtain a facial feature vector, and matching the target face image based on the facial feature vector. The historical face image optimization step includes adjusting the influencing parameters when selecting the standard angle face image through historical facial features.

[0125] The above-mentioned scene feature-based image face alignment device can extract background features from the target scene corresponding to the shooting camera using a pre-trained editing vector task model through the background feature conversion unit 61, and convert the background features into corresponding facial attribute editing vectors. The captured target facial image and the facial attribute editing vector are input into the pre-trained image alignment model through the face alignment unit 62, and a standard angle facial image corresponding to the target facial image is output. The image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical facial image optimization step. The initial matching step includes performing facial feature matching on the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector. The historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features. In this embodiment, the background features obtained by the camera can be converted into facial attribute editing vectors, and facial detail attribute features can be obtained to assist in face prediction. At the same time, the image alignment model is optimized through the facial historical feature image, thereby improving the accuracy of the model in matching standard angle facial images, thereby solving the technical problems of finding key points for standard angle mapping, missing detail features, and image face recognition accuracy when performing face alignment in related technologies.

[0126] Optionally, when performing initial matching, the image face alignment device based on scene features includes: a feature matching unit, which is used to use a fully convolutional network to perform feature matching on the captured historical face image to generate a face feature vector, wherein the angle difference between the current angle of the historical face image and the frontal angle of the standard angle face image is less than a preset angle difference threshold; a first feature vector input unit, which is used to input the face feature vector into the fully connected layer to obtain the network input parameters of the generator in the adversarial network, and combine the pre-input noise vector to output the predicted standard angle face image by the generator, wherein the network input parameters pass through multiple fully connected layers in the adversarial network generator to obtain the W space vector, and the W space vector is combined with the pre-input noise; a first face image input unit, which is used to input the predicted standard angle face image into the discriminator of the adversarial network, and after the discriminator performs downsampling processing, it is combined with the face feature vector to output a probability value, wherein the probability value is used to characterize whether the predicted standard angle face image is a successful matching image or a failed matching image corresponding to the historical face image.

[0127] Optionally, when training the discriminator, the image face alignment device based on scene features includes: a sample receiving unit for receiving positive samples and negative samples, wherein the positive samples are real standard-angle face images and their corresponding face feature vectors, and the negative samples include two types: a combination of unmatched real face feature vectors and standard-angle face images, and an image generated by a generator based on a random noise vector and a processed face feature vector; a discriminator training unit for updating the parameters of the discriminator based on positive samples and negative samples through an alternating training strategy, so that the discriminator distinguishes between real images and generated images, and distinguishes whether the face feature vector and the face image match; an optimization unit for mutually optimizing the generator and the discriminator through adversarial learning during the training process, wherein the goal of the generator is to increase the recognition rate of the discriminator for the generated images, and the goal of the discriminator is to reduce the recognition rate of the generated images and improve the recognition rate of the real images and matching feature vectors.

[0128] Optionally, the background feature conversion unit includes: a background feature vector extraction module for extracting a background feature vector from an input scene image using a convolutional neural network, where the background feature vector includes at least one of the following: the geometric layout of the scene, lighting conditions, multiple character postures, and character positions; a feature vector fusion module for fusing the background feature vector with the face feature vector, processing the background feature vector, the face feature vector, and the vector difference through a fully connected layer to obtain a first vector, wherein the vector difference is the difference between the background feature vector and the face feature vector, and the first vector is used to characterize the difference between the background feature vector and the face feature vector; a vector interaction module for interactively processing the first vector and the face feature vector using an attention mechanism to obtain a second vector, wherein the second vector is used to characterize the correlation between the background feature vector and the face feature vector; a vector input module for inputting the second vector into a fully connected layer, limiting the range through a preset activation function, adding it to the first attribute editing vector, and then again through the preset activation function to generate a face attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the background feature vector is input into the StyleFlow model.

[0129] Optionally, the scene feature-based image face alignment device also includes: a loss function calculation unit, which is used to calculate the loss function based on the face attribute editing vector and the second attribute editing vector after generating the face attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the face feature vector is input into the StyleFlow model; a model updating unit, which is used to adjust the parameters of the editing vector task model based on the loss function and update the editing vector task model.

[0130] Optionally, when optimizing historical face images, the image face alignment device based on scene features includes: a first picture selection unit, which is used to select pictures containing face videos from the input samples for the first training stage of the image alignment model to obtain a historical image set, obtain a common feature vector of all historical faces based on the historical image set, use the common feature vector to replace random noise, input it to the generator of the adversarial network, and the generator outputs a standard angle face image corresponding to the historical face image; a second picture selection unit, which is used to randomly select any image from the historical image set for the second training stage of the image alignment model, input random noise and any selected image into the generator of the adversarial network, and the generator outputs a standard angle face image corresponding to any image; a model parameter updating unit, which is used to alternately update the parameters of the image alignment model and freeze some fully connected layers during the first training stage and the second training stage of the image alignment model, and adjust the dependence of the image alignment model on key facial features and common feature vectors; a loss function guidance unit, which is used to use a predetermined loss function to guide the training of the generator and discriminator in the adversarial network during the first training stage and the second training stage of the image alignment model.

[0131] Optionally, the first image selection unit includes: an image set input module, used to input the historical image set into a fully convolutional network, use the fully convolutional network to perform feature matching on historical facial images in the historical image set, and generate a facial feature vector; a key feature splicing module, used to input the facial feature vector into a fully connected layer to obtain facial key features, and splice the facial key features, and input the spliced features into multiple convolutional layers to obtain a common feature vector of all historical faces.

[0132] The above-mentioned image face alignment device based on scene features may also include a processor and a memory. The above-mentioned background feature conversion unit 61, face alignment unit 62, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0133] The processor includes a kernel that retrieves the corresponding program unit from the memory. One or more kernels can be configured, and kernel parameters can be adjusted to achieve facial image alignment based on fixed camera scene features and historical facial features.

[0134] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0135] Example 3

[0136] An embodiment of the present application may provide an electronic device, Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 Only one is shown) processor 702, memory 704, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0137] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the scene feature-based image face alignment method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned scene feature-based image face alignment method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0138] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: using a pre-trained editing vector task model to extract background features from the target scene corresponding to the shooting camera, and converting the background features into corresponding facial attribute editing vectors; inputting the captured target facial image and the facial attribute editing vector into a pre-trained image alignment model, and outputting a standard angle facial image corresponding to the target facial image, wherein the image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical facial image optimization step. The initial matching step includes performing facial feature matching on the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector. The historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features.

[0139] It can be understood by those skilled in the art that Figure 7 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), or a PAD. Figure 7 It does not limit the structure of the above electronic device. For example, the electronic device may also include Figure 7 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 7 Different configurations shown.

[0140] Those skilled in the art will understand that all or part of the steps in the various scene feature-based image face alignment methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program. The program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0141] Example 4

[0142] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the scene feature-based image face alignment method provided in the first embodiment.

[0143] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the image face alignment method based on scene features of any one of the above-mentioned embodiments.

[0144] Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0145] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image face alignment method based on scene features described in each embodiment of the present application.

[0146] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the image face alignment method based on scene features described in each embodiment of the present application.

[0147] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0148] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0151] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0153] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for image face alignment based on scene features, characterized in that: include: A pre-trained edit vector task model is used to extract background features from the target scene corresponding to the shooting camera, and the background features are converted into corresponding face attribute edit vectors; The captured target facial image and the facial attribute editing vector are input into a pre-trained image alignment model, and a standard angle facial image corresponding to the target facial image is output, wherein the image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical facial image optimization step. The initial matching step includes performing facial feature matching on the captured target facial image to obtain a facial feature vector, and matching the target facial image based on the facial feature vector. The historical facial image optimization step includes adjusting the influencing parameters when selecting the standard angle facial image through historical facial features.

2. The image face alignment method according to claim 1, characterized in that The initial matching step includes: Using a fully convolutional network to perform feature matching on the captured historical facial image to generate a facial feature vector, wherein the angle difference between the current angle of the historical facial image and the frontal angle of the standard angle facial image is less than a preset angle difference threshold; Inputting the facial feature vector into a fully connected layer to obtain network input parameters of a generator in an adversarial network, combining the pre-input noise vector, and having the generator output a predicted standard angle face image, wherein the network input parameters pass through multiple fully connected layers in the adversarial network generator to obtain a W space vector, and the W space vector is combined with the pre-input noise; The predicted standard angle face image is input into the discriminator of the adversarial network. After downsampling processing, the discriminator combines the facial feature vector to output a probability value, wherein the probability value is used to characterize whether the predicted standard angle face image is a successful matching image or a failed matching image corresponding to the historical face image.

3. The image face alignment method according to claim 2, characterized in that: The training method of the discriminator includes: Receive positive samples and negative samples, wherein the positive samples are real standard-angle face images and their corresponding facial feature vectors, and the negative samples include two types: a combination of unmatched real face feature vectors and standard-angle face images, and an image generated by a generator based on a random noise vector and a processed facial feature vector; Updating the parameters of the discriminator based on the positive samples and the negative samples through an alternating training strategy so that the discriminator distinguishes between real images and generated images, and distinguishes whether the facial feature vector matches the facial image; During the training process, the generator and the discriminator optimize each other through adversarial learning. The goal of the generator is to increase the recognition rate of the discriminator for generated images, and the goal of the discriminator is to reduce the recognition rate of generated images and improve the recognition rate of real images and matching feature vectors.

4. The image face alignment method according to claim 1, characterized in that The step of converting the background features into corresponding face attribute editing vectors includes: A convolutional neural network is used to extract a background feature vector from an input scene image, wherein the background feature vector includes at least one of the following: the geometric layout of the scene, the lighting conditions, and the postures and positions of various characters; fusing the background feature vector with the face feature vector, processing the background feature vector, the face feature vector, and a vector difference through a fully connected layer to obtain a first vector, wherein the vector difference is the difference between the background feature vector and the face feature vector, and the first vector is used to represent the difference between the background feature vector and the face feature vector; Using an attention mechanism to interactively process the first vector and the facial feature vector to obtain a second vector, wherein the second vector is used to represent the correlation between the background feature vector and the facial feature vector; The second vector is input into the fully connected layer, the range is limited by the preset activation function, and it is added to the first attribute editing vector, and then the preset activation function is used again to generate a face attribute editing vector, wherein the first attribute editing vector is obtained by the image attribute classification module of the StyleFlow model after the background feature vector is input into the StyleFlow model.

5. The image face alignment method according to claim 4, characterized in that: After generating the face attribute edit vector, it also includes: Calculating a loss function based on the facial attribute edit vector and a second attribute edit vector, wherein the first attribute edit vector is obtained by an image attribute classification module of the StyleFlow model after inputting the facial feature vector into the StyleFlow model; Parameters of the edit vector task model are adjusted based on the loss function to update the edit vector task model.

6. The image face alignment method according to claim 1, characterized in that: The historical face image optimization steps include: In the first training phase of the image alignment model, images containing facial videos are selected from the input samples to obtain a historical image set. A common feature vector of all historical faces is obtained based on the historical image set. The common feature vector is used to replace random noise and is input to the generator of the adversarial network. The generator then outputs a standard-angle facial image corresponding to the historical facial image. In the second training phase of the image alignment model, any image from the historical image set is randomly selected, random noise and the selected image are input into a generator of the adversarial network, and the generator outputs a standard-angle face image corresponding to the image; During the first training phase and the second training phase of the image alignment model, alternately updating the parameters of the image alignment model and freezing some fully connected layers to adjust the dependence of the image alignment model on key facial features and the common feature vector; During the first training phase and the second training phase of the image alignment model, a predetermined loss function is used to guide the training of the generator and the discriminator in the adversarial network.

7. The image face alignment method according to claim 6, characterized in that: The step of obtaining a common feature vector of all historical faces based on the historical image set includes: Inputting the historical image set into a fully convolutional network, performing feature matching on historical facial images in the historical image set using the fully convolutional network to generate facial feature vectors; The facial feature vector is input into a fully connected layer to obtain key facial features, and the key facial features are spliced. The spliced features are input into multiple convolutional layers to obtain a common feature vector of all historical faces.

8. An image face alignment device based on scene features, characterized in that: include: A background feature conversion unit is used to extract background features from a target scene corresponding to the shooting camera using a pre-trained edit vector task model, and convert the background features into corresponding facial attribute edit vectors; A face alignment unit is used to input the captured target face image and the face attribute editing vector into a pre-trained image alignment model, and output a standard angle face image corresponding to the target face image, wherein the image alignment model includes a generator and a discriminator of an adversarial network. When training the image alignment model, it includes an initial matching step and a historical face image optimization step. The initial matching step includes performing facial feature matching on the captured target face image to obtain a facial feature vector, and matching the target face image based on the facial feature vector. The historical face image optimization step includes adjusting the influencing parameters when selecting the standard angle face image through historical facial features.

9. An electronic device, characterized in that: The method comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the scene feature-based image face alignment method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image face alignment method based on scene features described in any one of claims 1 to 7 are implemented.

Images