Face editing data enhancement method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a face editing data enhancement method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and computer vision, and the method comprises the steps: obtaining a face base map, obtaining a face detection frame and face key points, obtaining a face feature vector of the face base map through a face recognition algorithm, and obtaining the face feature vector of the face base map; adding image description to the face base map, using a generation diffusion large model to finely adjust the face base map according to the image description and then initializing an image editing model, setting editing parameter boundary values according to the face detection frame and the face key points, and setting editing position parameters according to the editing parameter boundary values to generate an editing face map. And performing feature similarity matching on the face feature vector and the editing feature vector to obtain a new face base map. According to the method, the face image is finely adjusted by utilizing the generated diffusion model, so that highly personalized image editing is realized, and the problem that the difference between the face data generated in the prior art and the real face is large is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a method, device, electronic device and storage medium for enhancing face editing data. Background Art

[0002] As an important research field of artificial intelligence, face recognition has a wide range of application scenarios, especially in terminal devices such as access control systems and mobile phone unlocking. However, face recognition still faces many challenges in practical applications, especially the significant impact of face posture changes on recognition accuracy. Traditional face recognition methods usually require high-definition, non-deflected face images facing the camera, which greatly limits the application of face recognition in complex scenarios such as security monitoring.

[0003] In recent years, with the vigorous development of large model generation technology, especially the rise of algorithms such as GAN (Generative Adversarial Network) and Diffusion (Diffusion Model), image generation and editing technology has made significant progress. These technologies can not only generate high-quality synthetic images in the text-to-image field, but also achieve fine control in real image editing, maintaining the target characteristics while changing specific elements in the image. This provides a new idea for solving the problem of posture change in face recognition.

[0004] However, current face recognition technology is still insufficient when dealing with multi-pose faces. Traditional solutions, such as optimizing the model backbone network, designing a more reasonable loss function, filtering or generating structured data, etc., have improved the recognition effect to a certain extent, but the effect is limited and difficult to popularize in practical applications. In particular, purely synthetic face images are quite different from real faces, and the effect on model training is limited.

[0005] Therefore, there is a need for a face editing data enhancement method that can generate high-quality, diverse multi-pose face images and improve the face recognition effect and accuracy in complex scenes. Summary of the invention

[0006] The embodiments of the present invention provide a method for enhancing face editing data to solve the problem that the face data generated by the prior art is very different from the real face and cannot be used for face recognition model training and comparison. The technical solution is as follows:

[0007] According to one aspect of the present invention, a method for enhancing face editing data includes: obtaining a face base map, performing face detection on the face base map using a face detection algorithm to obtain a face detection frame and face key points, and obtaining a face feature vector of the face base map through a face recognition algorithm; adding an image description to the face base map, using a generative diffusion macro model to fine-tune the face base map according to the image description to obtain a fine-tuning weight and then initialize an image editing model; setting editing parameter boundary values ​​according to the face detection frame and face key points, setting editing position parameters according to the editing parameter boundary values, and generating an edited face map according to the editing position parameters and the image editing model; obtaining an editing feature vector of the edited face map through a face recognition algorithm, and performing feature similarity matching on the face feature vector and the editing feature vector to obtain a new face base map.

[0008] In one of the embodiments, obtaining the facial feature vector of the facial base image through a facial recognition algorithm is achieved by the following steps: processing the facial base image according to the face detection frame to obtain a face, performing affine correction on the face according to the facial key points, and performing feature extraction on the corrected face using a face recognition algorithm to obtain a facial feature vector; the dimension of the feature vector is 512.

[0009] In one of the embodiments, the image editing model is initialized after fine-tuning the face base map according to the image description to obtain fine-tuning weights by using the generative diffusion model through the following steps: using a marker to tokenize the image description to obtain tags, and using an embedding model to convert the tags to a high-dimensional space to obtain prompt embeddings; using a model fine-tuning algorithm to initialize the feature layer in the generative diffusion model, and using a variational autoencoder in the generative large model to extract features from the face base map to obtain a latent vector; using a denoising diffusion implicit model to invert according to the latent vector and the prompt embedding to obtain predicted noise; the model fine-tuning algorithm is the LoRA algorithm; the denoising diffusion implicit model is the DDIM algorithm; randomly generating a noise vector of the same dimension as the latent vector, calculating the loss value according to the predicted noise and the noise vector and performing gradient back-propagation optimization to obtain fine-tuning weights, and initializing the image editing model according to the weights of the generative diffusion model and the fine-tuning weights; the image editing model is a DragDoise model.

[0010] In one of the embodiments, setting the editing parameter boundary values ​​according to the face detection frame and the face key points is achieved by the following steps: obtaining the editing parameter boundary values ​​of the horizontal coordinates of the face according to the range of deflection between the left eye and the right eye in the face detection frame; drawing concentric circles according to the face key points and calculating the moving length of the face key points based on the same rotation arc to obtain the editing parameter boundary values ​​of the vertical coordinates of the face; obtaining the editing parameter boundary values ​​of the vertical coordinates of the face according to the positions of the midpoints of the eyes and the nose in the face key points.

[0011] In one embodiment, setting the editing position parameters according to the editing parameter boundary values ​​is achieved by the following steps: setting multiple groups of editing position parameters of the horizontal face coordinates according to the editing parameter boundary values ​​of the horizontal face coordinates, setting multiple groups of editing position parameters of the vertical face coordinates according to the editing parameter boundary values ​​of the vertical face coordinates, and setting multiple groups of editing position parameters of the vertical face coordinates according to the editing parameter boundary values ​​of the vertical face coordinates.

[0012] In one of the embodiments, generating an edited face image based on the editing position parameters and the image editing model is achieved by the following steps: randomly combining the editing position parameters of the horizontal coordinate of the face, the editing position parameters of the vertical coordinate of the face, and the editing position parameters of the vertical coordinate of the face to obtain multiple groups of two-dimensional coordinates, and performing vector operations on the two-dimensional coordinates as parameters of the image editing model to generate an edited face image.

[0013] In one of the embodiments, feature similarity matching of the facial feature vector and the edited feature vector to obtain a new facial base map is achieved by the following steps: feature similarity matching of the facial feature vector and the edited feature vector, if the match is successful, using the edited facial map as the new facial base map; if the match is unsuccessful, adjusting the editing parameter boundary value, regenerating the editing position parameter according to the adjusted editing parameter boundary value, and regenerating the edited facial map according to the regenerated editing position parameter until the match is successful.

[0014] According to one aspect of the present invention, a face editing data enhancement device comprises: a feature extraction module, used to obtain a face base map, use a face detection algorithm to perform face detection on the face base map to obtain a face detection frame and face key points, and obtain a face feature vector of the face base map through a face recognition algorithm; a model initialization module, used to add an image description to the face base map, use a generative diffusion macro model to fine-tune the face base map according to the image description to obtain a fine-tuning weight and then initialize an image editing model; an editing data acquisition module, used to set editing parameter boundary values ​​according to the face detection frame and face key points, set editing position parameters according to the editing parameter boundary values, and generate an edited face map according to the editing position parameters and the image editing model; an editing data enhancement module, used to obtain an editing feature vector of the edited face map through a face recognition algorithm, and perform feature similarity matching on the face feature vector and the editing feature vector to obtain a new face base map.

[0015] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the face editing data enhancement method as described above.

[0016] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the face editing data enhancement method as described above.

[0017] The beneficial effects brought by the technical solution provided by the present invention are:

[0018] In the above technical scheme, the present invention first obtains a face base map, uses a face detection algorithm to perform face detection on the face base map to obtain a face detection frame and face key points, obtains a face feature vector of the face base map through a face recognition algorithm, adds an image description to the face base map, uses a generated diffusion macro model to fine-tune the face base map according to the image description to obtain a fine-tuning weight and then initializes the image editing model, sets editing parameter boundary values ​​according to the face detection frame and face key points, sets editing position parameters according to the editing parameter boundary values, generates an edited face map according to the editing position parameters and the image editing model, and obtains the edited face through a face recognition algorithm. The editing feature vector of the face image is matched with the editing feature vector by feature similarity to obtain a new face base image. By integrating face detection, recognition and image generation technology, highly personalized image editing is achieved. The advanced generative diffusion model is used to fine-tune the face image, which not only maintains the naturalness of the original image, but also incorporates the user's intention. It not only enhances the flexibility and creativity of image editing, but also ensures the consistency and realism before and after editing through feature similarity matching, thereby effectively solving the problem that the face data generated by the existing technology is very different from the real face and cannot be used for face recognition model training and comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0020] Figure 1 is a flow chart of a method for enhancing face editing data according to an exemplary embodiment;

[0021] Figure 2 It is a flowchart of a face editing data enhancement method in an application scenario;

[0022] Figure 3 is a block diagram of a face editing data enhancement device according to an exemplary embodiment;

[0023] Figure 4 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0024] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0026] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0027] The present invention provides a face editing data enhancement method, which realizes highly personalized image editing by integrating face detection, recognition and image generation technology, solves the problem that the face data generated by the prior art is greatly different from the real face and cannot be used for face recognition model training and comparison, and the face editing data enhancement method is suitable for a face editing data enhancement device, which can be an electronic device. The face editing data enhancement method in the embodiment of the present invention can be applied to a variety of scenarios, such as face editing data enhancement, etc.

[0028] See also Figure 1 An embodiment of the present invention provides a method for enhancing face editing data, which is applicable to electronic devices.

[0029] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0030] like Figure 1 As shown, the method may include the following steps:

[0031] Step 110, obtain a face base map, use a face detection algorithm to perform face detection on the face base map to obtain a face detection frame and face key points, and obtain a face feature vector of the face base map through a face recognition algorithm.

[0032] In a possible implementation, a face base image is processed according to a face detection frame to obtain a face, an affine correction is performed on the face according to the face key points, and a face recognition algorithm is used to extract features of the corrected face to obtain a face feature vector.

[0033] Among them, the dimension of the feature vector is 512.

[0034] Step 130, adding an image description to the face base map, using the generated diffusion model to fine-tune the face base map according to the image description to obtain the fine-tuning weights and then initialize the image editing model.

[0035] In one possible implementation, a tagger is used to tokenize the image description to obtain tags, an embedding model is used to convert the tags to a high-dimensional space to obtain a hint embedding, a model fine-tuning algorithm is used to initialize the feature layer in the generative diffusion model, a variational autoencoder in the generative model is used to extract features from the face base map to obtain a latent vector, a denoising diffusion implicit model is used to invert the latent vector and the hint embedding to obtain predicted noise, a noise vector of the same dimension as the latent vector is randomly generated, a loss value is calculated based on the predicted noise and the noise vector, and gradient back-propagation optimization is performed to obtain fine-tuning weights, and the image editing model is initialized based on the weights of the generative diffusion model and the fine-tuning weights.

[0036] Among them, the model fine-tuning algorithm is the LoRA algorithm, the denoising diffusion implicit model is the DDIM algorithm, and the image editing model is the DragDoise model.

[0037] Among them, the LoRA (Low-Rank Adaptation) algorithm is a technology used to optimize deep learning models, especially when dealing with large-scale pre-trained models. Its core idea is to introduce a low-rank matrix to approximate the weight update of the model, thereby achieving rapid adjustment and adaptation of the model without significantly increasing the computational and storage costs; the DragNoise model is an interactive point-controlled image editing method based on the semantic propagation of the diffusion model. The core idea of ​​this model is to use the "middle-block replacement" operation in the diffusion model to achieve efficient and accurate image editing by optimizing and propagating bottleneck features. The advantage of the DragNoise model is that it can effectively retain the original information of the image while achieving more accurate and fast drag editing. Compared with the existing SOTA method DragDiffusion, DragNoise significantly reduces more than 50% of the optimization steps, and can also achieve accurate control in extreme cases where the features around the editing point are similar. In addition, DragNoise performs well in content filling, erasing, face turning, posture control, etc., meeting the diverse needs of users for image editing.

[0038] Step 150, setting editing parameter boundary values ​​according to the face detection frame and the face key points, setting editing position parameters according to the editing parameter boundary values, and generating an edited face image according to the editing position parameters and the image editing model.

[0039] In one possible implementation, the editing parameter boundary value of the horizontal coordinate of the face is obtained according to the deflection range between the left eye and the right eye in the face detection frame, the editing parameter boundary value of the vertical coordinate of the face is obtained by drawing concentric circles according to the face key points and calculating the moving length of the face key points based on the same rotation arc, and the editing parameter boundary value of the vertical coordinate of the face is obtained according to the position of the midpoint of the eyes and the position of the nose in the face key points.

[0040] In one possible implementation, multiple groups of editing position parameters of the horizontal face coordinates are set according to the editing parameter boundary values ​​of the horizontal face coordinates, multiple groups of editing position parameters of the vertical face coordinates are set according to the editing parameter boundary values ​​of the vertical face coordinates, and multiple groups of editing position parameters of the vertical face coordinates are set according to the editing parameter boundary values ​​of the vertical face coordinates.

[0041] In one possible implementation, the editing position parameters of the horizontal coordinate of the face, the editing position parameters of the vertical coordinate of the face, and the editing position parameters of the vertical coordinate of the face are randomly combined to obtain multiple sets of two-dimensional coordinates, and vector operations are performed on the two-dimensional coordinates as parameters of the image editing model to generate an edited face image.

[0042] Specifically, the three Euler angles of Yaw, Pitch and Roll are used to define the facial posture, which correspond to deflection, pitch and rotation respectively. The parameter boundary values ​​of Yaw, Roll and Pitch are set to ensure that the edited face remains recognizable. Multiple sets of editing position parameters of Yaw, Roll and Pitch are designed, and they are randomly combined to form drag point parameters that meet the editing requirements.

[0043] Step 170, obtaining an edited feature vector of the edited face image through a face recognition algorithm, performing feature similarity matching between the face feature vector and the edited feature vector to obtain a new face base image.

[0044] In one possible implementation, feature similarity matching is performed on the facial feature vector and the edited feature vector. If the match is successful, the edited face image is used as the new face base image. If the match is unsuccessful, the editing parameter boundary values ​​are adjusted, and the editing position parameters are regenerated according to the adjusted editing parameter boundary values. The edited face image is regenerated according to the regenerated editing position parameters until the match is successful.

[0045] In one possible implementation, the face editing algorithm may include DragGAN, FreeDrag, DragDiffusion, DragNoise and other generation algorithms based on GAN or Diffusion, the face detection algorithm may include general algorithms such as mtcnn, retinaface, tinaface, or training target detection algorithms such as yolo, and the face recognition algorithm may include arcface, adaface and other feature extraction and feature matching algorithms, which are not limited here.

[0046] Through the above process, the present invention first obtains a face base map, uses a face detection algorithm to perform face detection on the face base map to obtain a face detection frame and face key points, obtains a face feature vector of the face base map through a face recognition algorithm, adds an image description to the face base map, uses a generated diffusion macro model to fine-tune the face base map according to the image description to obtain a fine-tuning weight, and then initializes the image editing model, sets the editing parameter boundary value according to the face detection frame and the face key points, sets the editing position parameter according to the editing parameter boundary value, generates an edited face map according to the editing position parameter and the image editing model, and obtains the edited face through a face recognition algorithm. The face feature vector of the image is edited, and the face feature vector and the edited feature vector are matched for feature similarity to obtain a new face base map. By integrating face detection, recognition and image generation technology, highly personalized image editing is achieved, and the advanced generative diffusion model is used to fine-tune the face image, which not only maintains the naturalness of the original image, but also incorporates the user's intention. It not only enhances the flexibility and creativity of image editing, but also ensures the consistency and realism before and after editing through feature similarity matching, thereby effectively solving the problem that the face data generated by the existing technology is very different from the real face and cannot be used for face recognition model training and comparison.

[0047] In one application scenario, Figure 2 A flowchart of a face editing data augmentation method is shown.

[0048] like Figure 2 As shown in the figure, first a face base image is read from the data source as the starting input of the entire processing flow, the StableDiffusion model is initialized, the basic environment is prepared for the model, and general description prompts are added to the face base image. These prompts will guide the model to understand and generate image content that matches the description. The image is processed using a face detector to detect and extract the face's bounding box (bbox) and key points (landmark). This information will be used for subsequent image editing and correction.

[0049] Furthermore, the StableDiffusion model is fine-tuned with LoRA to generate LoRA weights (lora_weights), and the StableDiffusion model and LoRA weights (lora_weights) are used to initialize the DragNOise model to help the model optimize performance for specific tasks (such as face editing).

[0050] Furthermore, bbox is used to perform face cutout operation to separate the face from the background. At the same time, landmark is used to perform face affine correction to ensure that the cutout face is consistent with the original image in shape and angle. The face recognition algorithm is used to extract the face feature vector (feat_base_0), which will be used for subsequent face recognition tasks.

[0051] Furthermore, editing boundary points and multiple sets of position parameters (xi, yi) are set for the face base image, which define the specific position and range of image editing. For each set of position parameters, the DragNoise model or other editing tools are used to generate the edited image (lmage_i).

[0052] Finally, face detection, bbox and landmark extraction are performed again on the edited image, and the extracted feature vector (feat_base_i) is compared with the original feature vector (feat_base_0) for face similarity. If the match is successful, the edited image is stored in the database or processed subsequently. If there is no match, the image is discarded or the editing operation is performed again.

[0053] Through the above process, the embodiment of the present invention realizes an efficient and accurate face editing and recognition process by deeply integrating the StableDiffusion model, LoRA weight fine-tuning and DragNoise model, which not only simplifies the complex process of traditional image editing, but also greatly enhances the controllability and diversity of image content through the introduction of the prompt word Prompt. The application of LoRA weights enables the model to be quickly optimized for specific tasks (such as face editing), which significantly improves processing efficiency and effect. In addition, face detection, bounding box extraction and key point correction technology ensure the integrity and accuracy of the face structure during the editing process, and effectively avoid the distortion problem of the edited image. The application of face recognition algorithm provides a reliable similarity comparison basis for images before and after editing, and ensures the rationality and consistency of the editing results.

[0054] In summary, the embodiments of the present invention achieve efficient, accurate and controllable face editing and recognition through advanced technology integration and innovation, bringing a new solution to the field of image processing and helping to improve the user experience and effects of related applications.

[0055] In another application scenario, a face editing data enhancement method may include the following steps:

[0056] Step S1, select a face base map from the data source as input, use the face detection algorithm to detect the face detection box bbox (upper left point (x1, y1), lower right point (x2, y2)), and extract the face key points landmarks (left eye (lex, ley), right eye (rex, rey), nose (nx, ny), left corner of the mouth (lmx, lmy), right corner of the mouth (rmx, rmy)).

[0057] Step S2, cut out the face according to the bbox, use landmarks to perform affine correction on the face to ensure the standardization of the face shape and angle, and extract the 512-dimensional feature vector feat_base_0 from the corrected face through face recognition algorithms such as AdaFace.

[0058] Step S3, adding a general image description prompt to the face base image, such as "identification photo, asian man, age 30, black hair, frontal".

[0059] Step S4: Use the tokenizer to tokenize Prompt and convert it into prompt_embeds in a high-dimensional space. Initialize the UNET module feature layer in Stable Diffusion according to the LoRA algorithm, and use the VAE module to extract the initial latent vector of the face base map.

[0060] Step S5, perform Inversion based on the initial latent vector and prompt_embeds through the DDIM algorithm to obtain the predicted noise noise_pred, randomly generate a noise vector with the same dimension as the initial latent, calculate the loss with noise_pred, and optimize the LoRA module, repeat the iterative optimization until the LoRA fine-tuning is completed, and obtain lora_weight.

[0061] Step S6, use the Stable Diffusion model weights and lora_weight to jointly initialize the DragNoise model.

[0062] Step S7, set the editing parameter boundary points according to the bbox and landmarks, and define the deflection / rotation / pitch corresponding to the Euler angle Yaw / Roll / Pitch to describe the facial posture. That is, for a frontal face photo, with the horizontal and vertical directions as XY and the vertical direction as Z, rotating the face around the Y axis is Yaw, rotating around the X axis is Pitch, and rotating around the Z axis is Roll.

[0063] Specifically, set the boundary values ​​of Yaw parameters: the horizontal coordinates xstart, xend of the starting and ending points of Yaw deflection should be between the left and right eyes, and the boundary value of the horizontal coordinate is (lex, rex); the vertical coordinate can be within the range of the face, that is, the boundary value of the vertical coordinate is (y1, y2); the vertical coordinate of the starting point must remain consistent.

[0064] Set the parameter boundary value of Roll: draw concentric circles with the center point of the nose at the left eye, right eye, left corner of the mouth, and right corner of the mouth, with each point as the starting point. Calculate the drag length based on the same rotation arc along the tangent direction of the concentric circle. The rotation arc should be between 0-90°.

[0065] Set the parameter boundary values ​​of Pitch: the starting point of Pitch is the midpoint of the left and right eyes, the horizontal coordinate of the end point is the same as the starting point, and the vertical coordinate boundary is about 1 / 3 above the nose.

[0066] Furthermore, multiple groups of Yaw editing position parameters are set: randomly select several starting horizontal coordinates yaw_x at (lex,nx), and select the corresponding end horizontal coordinates at (yaw_x,rex) to obtain the left turn parameter group, and vice versa, select several right turn parameter groups. The yaw_y value can be the vertical coordinate of the midpoint of the two eyes (ley+rey) / 2.

[0067] Set multiple sets of Roll editing position parameters: take a number of values ​​between (0,60°) as the rotation angle in the clockwise / counterclockwise direction of the concentric circle, and calculate the corresponding start and end point coordinate parameter sets based on the concentric circle radius and tangent.

[0068] Set multiple sets of Pitch editing position parameters: take the nose ny as the starting point, and take several random values ​​from the nose to the midpoint of the two eyes (ny, (ley+rey) / 2) as the end point, which corresponds to upward tilt; and take several random values ​​from the nose to the midpoint of the two mouth corners (lmy, rmy) / 2, ny) as the end point, which corresponds to downward tilt, with the nose position as the horizontal coordinate.

[0069] Step S8, randomly combine multiple sets of Yaw, Roll, and Pitch editing position parameters. Each set of Yaw / Roll / Pitch parameters is finally a coordinate (x, y) in a two-dimensional space. Randomly combine Yaw / Roll / Pitch to form a suitable editing position and then perform vector operations on the corresponding (x, y) to obtain multiple sets of positions.

[0070] Step S9, select a set of editing position parameters to input into the DragNoise model to generate the corresponding edited face image, perform face detection and key point extraction on the edited face image, perform cutout correction on the edited face image, and then extract a 512-dimensional feature vector feat_base_i by the face recognition algorithm.

[0071] Step S10, compare the feature similarity of feat_base_0 and feat_base_i. If the match is successful, the edited face is stored in the face base library with the same ID. Otherwise, it is discarded and the editing position boundary value is adjusted, and the editing position parameters are regenerated. Steps S9 to S10 are repeated to traverse all the set Yaw / Pitch / Roll editing position parameters, and a specified number of multi-pose face base maps are generated for each face base map. All successfully matched edited face images are integrated and stored in the corresponding face base library for subsequent use and management.

[0072] Through the above steps, the embodiment of the present invention realizes highly customized and precise face image generation and management by integrating advanced face detection, affine correction, LoRA fine-tuning, DragNoise editing and face recognition technology. By adding a descriptive prompt to the face base map and combining the LoRA fine-tuning Stable Diffusion model, the model can accurately generate face images that meet specific identities and attributes. The application of the DragNoise model further enriches the diversity of face postures and can generate multi-angle and multi-posture face images. It not only significantly improves the efficiency and accuracy of face image generation, but also ensures the identity consistency between the generated image and the original image through feature similarity comparison, effectively avoiding information distortion and confusion. In addition, the automated face image editing and management process greatly saves manpower and time costs, and provides high-quality data support for face recognition, security monitoring, virtual reality and other fields. Overall, the solution realizes the intelligence and refinement of face image generation and management through technological innovation.

[0073] The following is an embodiment of the device of the present invention, which can be used to perform the face editing data enhancement method involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the face editing data enhancement method involved in the present invention.

[0074] See also Figure 3 , an embodiment of the present invention provides a face editing data enhancement device 800.

[0075] The face editing data enhancement device 800 includes but is not limited to: a feature extraction module 810, a model initialization module 830, an editing data acquisition module 850 and an editing data enhancement module 870.

[0076] Among them, the feature extraction module 810 is used to obtain a face base map, use a face detection algorithm to perform face detection on the face base map to obtain a face detection frame and face key points, and obtain a face feature vector of the face base map through a face recognition algorithm.

[0077] The model initialization module 830 is used to add an image description to the face base map, use the generated diffusion model to fine-tune the face base map according to the image description to obtain the fine-tuning weights and then initialize the image editing model.

[0078] The editing data acquisition module 850 is used to set the editing parameter boundary value according to the face detection frame and the face key points, set the editing position parameter according to the editing parameter boundary value, and generate the editing face image according to the editing position parameter and the image editing model.

[0079] The edited data enhancement module 870 is used to obtain an edited feature vector of the edited face image through a face recognition algorithm, and perform feature similarity matching between the face feature vector and the edited feature vector to obtain a new face base image.

[0080] It should be noted that the face editing data enhancement provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the face editing data enhancement device will be divided into different functional modules to complete all or part of the functions described above.

[0081] In addition, the face editing data enhancement device and the face editing data enhancement method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.

[0082] Figure 4 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.

[0083] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 4 One or more components of exemplary electronic device 2000 are shown.

[0084] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 4 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0085] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0086] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this is not intended to be a specific limitation.

[0087] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0088] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0089] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 4 (not shown), each module may respectively contain computer-readable instructions for the electronic device 2000. For example, the face editing data enhancement device may be regarded as an application 253 deployed on the electronic device 2000.

[0090] The data 255 may be signal information, etc., stored in the memory 250 .

[0091] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the face editing data enhancement method is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0092] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0093] See also Figure 5In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.

[0094] exist Figure 5 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0095] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0097] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0098] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0099] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0100] The computer-readable instructions are executed by one or more processors 4001 to implement the face editing data enhancement method in the above-mentioned embodiments.

[0101] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the face editing data enhancement method as described above.

[0102] A computer program product is provided in an embodiment of the present invention. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the face editing data enhancement method as described above.

[0103] Compared with the related art, the present invention has the following beneficial effects:

[0104] 1. The present invention first obtains a face base map, uses a face detection algorithm to perform face detection on the face base map to obtain a face detection frame and face key points, obtains a face feature vector of the face base map through a face recognition algorithm, adds an image description to the face base map, uses a generated diffusion macro model to fine-tune the face base map according to the image description to obtain a fine-tuning weight, and then initializes the image editing model, sets an editing parameter boundary value according to the face detection frame and the face key points, sets an editing position parameter according to the editing parameter boundary value, generates an edited face map according to the editing position parameter and the image editing model, and obtains the edited face map through a face recognition algorithm. Edit the feature vector, perform feature similarity matching on the face feature vector and the edited feature vector to obtain a new face base map, and achieve highly personalized image editing by integrating face detection, recognition and image generation technology. Use the advanced generative diffusion model to fine-tune the face image, which not only maintains the naturalness of the original image, but also incorporates the user's intention. It not only enhances the flexibility and creativity of image editing, but also ensures the consistency and realism before and after editing through feature similarity matching, thereby effectively solving the problem that the face data generated by the existing technology is very different from the real face and cannot be used for face recognition model training and comparison.

[0105] 2. The present invention can improve the accuracy of face recognition in actual monitoring scenarios: by generating multi-pose face images, the face features in the base map database are enriched, so that the base map database can better match the multi-pose face states in actual use scenarios. This helps to improve the accuracy and robustness of the face recognition system in actual monitoring environments.

[0106] 3. The present invention can enhance data diversity: Traditional face recognition data enhancement methods often rely on simple transformations (such as rotation, scaling, etc.), which makes it difficult to generate diverse data. This solution can generate facial images with various changes in posture, expression, etc. through fine editing based on generating a large model, greatly enhancing the diversity of data.

[0107] 4. The present invention can maintain the consistency of face ID features: In the process of generating multi-pose face images, the present solution focuses on maintaining the consistency of face ID features, that is, the generated image is consistent with the original image in terms of face features. This helps ensure that during the face recognition process, the system can accurately identify faces in different poses.

[0108] 5. The present invention can improve the generalization ability of the model: since the base map database contains face images in more postures, the trained face recognition model will be able to better adapt to faces in different postures, thereby improving the generalization ability of the model.

[0109] 6. The present invention can reduce the reliance on high-quality frontal photos: Traditional face recognition systems often rely on high-quality frontal photos as base data. However, this solution can reduce the reliance on high-quality frontal photos to a certain extent by generating multi-pose face images, thereby reducing the difficulty and cost of data collection.

[0110] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0111] The above descriptions are only partial embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A face editing data enhancement method, characterized in that: The method comprises: Obtain a face base map, perform face detection on the face base map using a face detection algorithm to obtain a face detection frame and face key points, and obtain a face feature vector of the face base map using a face recognition algorithm; Add an image description to the face base map, use the generative diffusion model to fine-tune the face base map according to the image description to obtain a fine-tuning weight, and then initialize the image editing model; Setting editing parameter boundary values ​​according to the face detection frame and the face key points, setting editing position parameters according to the editing parameter boundary values, and generating an edited face image according to the editing position parameters and the image editing model; The edited feature vector of the edited face image is obtained by a face recognition algorithm, and feature similarity matching is performed on the face feature vector and the edited feature vector to obtain a new face base image.

2. The face editing data enhancement method according to claim 1, characterized in that: The step of obtaining the face feature vector of the face base image by using a face recognition algorithm includes: The face base map is processed according to the face detection frame to obtain a face, affine correction is performed on the face according to the face key points, and features are extracted from the corrected face using a face recognition algorithm to obtain a face feature vector; the dimension of the feature vector is 512.

3. The face editing data enhancement method according to claim 1, characterized in that: The method of using the generated diffusion model to fine-tune the face base map according to the image description to obtain a fine-tuning weight and then initializing the image editing model includes: Using a tagger to tokenize the image description to obtain tags, and using an embedding model to convert the tags into a high-dimensional space to obtain a prompt embedding; Initialize the feature layer in the generated diffusion model using a model fine-tuning algorithm, and use the variational autoencoder in the generated model to extract features from the face base image to obtain a latent vector; The denoising diffusion implicit model is used to invert the latent vector and the hint embedding to obtain the predicted noise; the model fine-tuning algorithm is the LoRA algorithm; the denoising diffusion implicit model is the DDI M algorithm; A noise vector of the same dimension as the latent vector is randomly generated, a loss value is calculated according to the predicted noise and the noise vector, and gradient back-propagation optimization is performed to obtain a fine-tuning weight, and an image editing model is initialized according to the weight of the generated diffusion model and the fine-tuning weight; the image editing model is a DragDoi se model.

4. The face editing data enhancement method according to claim 1, characterized in that: The step of setting the editing parameter boundary value according to the face detection frame and the face key points includes: Obtaining a boundary value of an editing parameter of a horizontal coordinate of a face according to a range of deflection between a left eye and a right eye in the face detection frame; Draw concentric circles according to the facial key points and calculate the moving length of the facial key points based on the same rotation arc to obtain the editing parameter boundary value of the vertical coordinate of the face; The editing parameter boundary value of the vertical coordinate of the face is obtained according to the position of the midpoint of the eye and the position of the nose in the face key points.

5. The face editing data enhancement method according to claim 4, characterized in that: The step of setting the editing position parameter according to the editing parameter boundary value comprises: The editing position parameters of multiple groups of face horizontal coordinates are set according to the editing parameter boundary values ​​of the face horizontal coordinates, the editing position parameters of multiple groups of face vertical coordinates are set according to the editing parameter boundary values ​​of the face vertical coordinates, and the editing position parameters of multiple groups of face vertical coordinates are set according to the editing parameter boundary values ​​of the face vertical coordinates.

6. The face editing data enhancement method according to claim 5, characterized in that: The step of generating an edited face image according to the edit position parameter and the image editing model comprises: The editing position parameters of the horizontal coordinate of the face, the editing position parameters of the vertical coordinate of the face and the editing position parameters of the vertical coordinate of the face are randomly combined to obtain multiple groups of two-dimensional coordinates, and vector operations are performed on the two-dimensional coordinates as parameters of the image editing model to generate an edited face image.

7. The face editing data enhancement method according to claim 1, characterized in that: The step of performing feature similarity matching on the face feature vector and the edited feature vector to obtain a new face base map includes: Perform feature similarity matching on the face feature vector and the edited feature vector, and if the match is successful, use the edited face image as a new face base image; If the match is unsuccessful, the editing parameter boundary value is adjusted, the editing position parameter is regenerated according to the adjusted editing parameter boundary value, and the editing face image is regenerated according to the regenerated editing position parameter until the match is successful.

8. A face editing data enhancement device, characterized in that: The device comprises: A feature extraction module is used to obtain a face base map, perform face detection on the face base map using a face detection algorithm to obtain a face detection frame and face key points, and obtain a face feature vector of the face base map using a face recognition algorithm; A model initialization module, used to add an image description to the face base map, use a generative diffusion model to fine-tune the face base map according to the image description to obtain a fine-tuning weight, and then initialize the image editing model; An editing data acquisition module, used to set editing parameter boundary values ​​according to the face detection frame and face key points, set editing position parameters according to the editing parameter boundary values, and generate an editing face image according to the editing position parameters and the image editing model; The editing data enhancement module is used to obtain the editing feature vector of the edited face image through a face recognition algorithm, and perform feature similarity matching on the face feature vector and the editing feature vector to obtain a new face base image.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the face editing data enhancement method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the face editing data enhancement method as described in any one of claims 1 to 7.

Images