Face shielding fusion method and system based on text control and readable storage medium
Through the face occlusion fusion method based on text control, high-quality face occlusion images are automatically synthesized using diffusion model and denoising network, which solves the insufficient accuracy and the need to synthesize images in the prior art face recognition model under occlusion situation. The problem of a lot of manual time has achieved efficient and automated image synthesis and data provision.
Patent Information
- Application Number
- CN202410059877.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-15
- Publication Date
- 2025-02-07
AI Technical Summary
The prior art is difficult to automatically synthesize high-quality face occlusion images, resulting in insufficient accuracy of the face recognition model in the case of occlusion, and the synthesis of images requires a lot of manual time.
Using a text-controlled face occlusion fusion method, by acquiring the foreground target image and background image, the foreground target is automatically fused to the lower half of the face occlusion area of the background image using a diffusion model and a denoising network.
Automatic image synthesis is realized, improving the accuracy of the face recognition model in occlusion situations, reducing manual operation time, and providing a rich source of data to improve model accuracy.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and more specifically, to a face occlusion fusion method, system and readable storage medium based on text control. Background Art
[0002] The accuracy requirements of learning models are getting higher and higher. The improvement of accuracy is achieved by adjusting the network model structure on the one hand, and by obtaining a large amount of training data on the other hand. However, due to the limitations of many factors in real scenes, the cost of obtaining data is getting higher and higher, resulting in the inability to obtain the amount of data we need in many scenes. Therefore, it is necessary for us to obtain the amount of data we need through synthesis to improve the accuracy of the model. Similarly, in order to improve the accuracy of the face recognition model when the face is occluded, it is necessary to collect a large number of unused face images with occluded faces. However, simply collecting complex scenes containing human targets on the Internet requires manual labor to spend a lot of time to select, or directly using PS software to directly synthesize human targets into complex scenes. The images synthesized by this method are the most accurate and realistic, but the synthesis of each image requires a lot of manual time. Summary of the invention
[0003] In view of the prior art, the technical problem solved by the present application is to provide a text-controlled face occlusion fusion method, a collection device and a recognition system that can automatically synthesize an occluded target onto a human face.
[0004] In order to solve the above technical problems, the present application provides a face occlusion fusion method based on text control, comprising:
[0005] Obtaining a foreground target image for occlusion;
[0006] Obtaining the lower half face occlusion area of the background image B containing the face; and,
[0007] Based on the lower face occlusion hint text and the diffusion model, the foreground target is fused to the lower face occlusion area of the background image B.
[0008] In one embodiment of the application, the step of obtaining the lower face occlusion area of the background image B containing the face includes:
[0009] Perform facial key point detection on the background image B to obtain the coordinates of three key points of the face: the center point of the left eye area (left.x, left.y), the center point of the right eye area (right.x, right.y) and the coordinates of the nose tip (nose.x, nose.y);
[0010] The lower face occlusion area is located according to the coordinates of the three key points: the coordinate range of the lower face occlusion area on the X-axis is (0, x.up) and the coordinate range on the Y-axis is (y.min, y.max);
[0011] Among them, left.x, left.y, right.x and right.y are the horizontal coordinates of the center point of the left eye area, the vertical coordinates of the center point of the left eye area, the horizontal coordinates of the center point of the right eye area, and the vertical coordinates of the center point of the right eye area respectively; y.min=min(right.y,left.y), y.max=y.min+(nose.yy.min)*2 / 5; x.up=|left.x-center.x|, center.x=(right.x-left.x) / 2; |.| means finding the absolute value, min(.) means taking the minimum value of the two, x.up is to obtain the maximum horizontal coordinate value of the lower half of the face occluded area on the X-axis, y.min and y.max are the minimum and maximum vertical coordinate values of the lower half of the face occluded area on the Y-axis respectively.
[0012] In one embodiment of the application, the foreground target is fused to the lower face occlusion area of the background image B based on the lower face occlusion prompt text and the diffusion model, including:
[0013] Get the encoded image of the foreground target image and the encoded image of the background image
[0014] Encoded image and the encoded image Fusion to obtain the foreground and background coded images
[0015] Text-driven diffusion model for encoding foreground and background images based on lower face occlusion hints De-noising and decoding are performed to obtain the fused image S with the lower half of the face occluded.
[0016] In one embodiment of the application, the background encoding image is encoded based on the lower half face occlusion prompt text driven diffusion model De-noising and decoding are performed to obtain the fused image S with the lower half of the face blocked, which is:
[0017] Encode the background image The lower half face occlusion prompt text information and foreground target information are input into the denoising network superimposed with the attention module of the diffusion model for iterative denoising to obtain the fused image S of the lower half face occlusion after denoising.
[0018] Among them, the lower half face occlusion prompt text information includes the face area in the background image B and the lower half face occlusion area of the background image B indicated by the prompt text information, and the foreground target information is the extracted mask image of the foreground target.
[0019] In one embodiment of the application, the diffusion model is based on the DDPM model, and the denoising network is based on the UNET network; the attention map obtained by inputting the attention module is:
[0020]
[0021]
[0022] Among them, t represents the time step of the denoising solution of the diffusion model to obtain the fused image S, and l represents the lth layer of the UNET network; and are respectively the spatial features of the foreground target image F and the background area B extracted by the network at layer l when the time step is t, and the background area B is the occluded area of the lower half of the face; and are respectively the query vector and key vector of the lth layer extracted when the attention module of the UNET network processes the background area B at time step t, and are the query vector and key vector of the lth layer extracted when the attention module of the UNET network processes the foreground target image F at time step t, and Respectively, vector and The transpose of d 1 and d 2 They are and The vector dimensions in and They are the self-attention map about the background image, the self-attention map about the foreground target image, and the self-attention map about and The attention diagram between the crosses; Soft max(.) indicates the normalization operation; V compose Constructing composite self-attention map based on regional pixel index The function factor of .
[0023] In one embodiment of the application, the coded image and the encoded image Fusion to obtain the foreground and background coded images for:
[0024]
[0025] Among them, z is Gaussian noise, the image area corresponding to the lower half of the face occlusion area of the background image B is called the main background area b, and X b The mask image of the main background area and the pixel value of the main background area in the mask image is 1, X F is the mask image corresponding to the foreground image fused on the occluded area of the lower half of the face, and the pixel value of the foreground target area in the mask image is 1. is the XOR result, and ⊙ is the XOR operation. T is equivalent to t, both indicating that the time is not long.
[0026] The present application provides a face fusion system, which includes:
[0027] An image acquisition unit, acquiring a foreground image and a background image; and,
[0028] An image processor is connected to the image acquisition unit and is used to execute the steps of the face occlusion and fusion method based on text control.
[0029] The present application also provides a computer-readable storage medium, which stores program code. When the computer program code runs on a computer, the computer executes the steps of the text-controlled face occlusion and fusion method.
[0030] The beneficial effects of the face occlusion fusion method, face fusion system and storage medium provided by the present application are: extracting the area to be fused that is occluded by the lower half of the face in the background image to be fused, automatically fusion the foreground target image to the occluded area of the lower half of the face in the background image, and obtaining a fused image in which the face is occluded by the object. The face occlusion fusion method based on text control can provide a rich data source for model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0032] Figure 1 This is a flowchart of the steps of the face occlusion fusion method based on text control in an embodiment of the present application;
[0033] Figure 2 A foreground image according to an embodiment of the present application;
[0034] Figure 3 The extracted background image of the embodiment of the present application;
[0035] Figure 4For the embodiments of this application Figure 2 and Figure 3 Schematic diagram of the fusion result. DETAILED DESCRIPTION
[0036] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0037] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.
[0038] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0040] The text-controlled face occlusion fusion method, system and readable storage medium of the present application are now described in detail with reference to the accompanying drawings.
[0041] Please refer to Figure 1 The face occlusion fusion method based on text control provided in the embodiment of the present application includes the following steps S100 to S300.
[0042] Step S100: obtaining a foreground target image for occlusion;
[0043] Step S200: obtaining the lower half face occlusion area of the background image B containing the face;
[0044] Step S300: Fusing the foreground target into the lower face occlusion area of the background image B based on the lower face occlusion prompt text and the diffusion model.
[0045] In one embodiment, image segmentation is performed in a foreground image to obtain an RGBA image of the foreground target area, that is, the foreground target image in step S100. Figure 2 As shown, it is a foreground image. Figure 2 The hat region in is the aforementioned foreground target image.
[0046] For step S200, obtaining the lower half face occlusion area of the background image B containing the face includes the following steps S210 and S220.
[0047] Step S210: perform facial key point detection on the background image B to obtain the coordinates of three key points of the face.
[0048] Among them, the three key points of the aforementioned face include the center point of the left eye area (left.x, left.y), the center point of the right eye area (right.x, right.y) and the coordinates of the nose tip (nose.x, nose.y);
[0049] Step S220: Locate the lower face occlusion area according to the coordinates of the three key points: the coordinate range of the lower face occlusion area on the X axis is (0, x.up) and the coordinate range on the Y axis is (y.min, y.max);
[0050] y.min=min(right.y,left.y);
[0051] y.max=y.min+(nose.yy.min)*2 / 5;
[0052] x.up = |left.x - center.x|;
[0053] center.x=(left.x-righ.x) / 2;
[0054] Among them, left.x, left.y, right.x and right.y are the horizontal coordinates of the center point of the left eye area, the vertical coordinates of the center point of the left eye area, the horizontal coordinates of the center point of the right eye area, and the vertical coordinates of the center point of the right eye area respectively; |.| means to find the absolute value, min(.) means to take the minimum value between the two, x.up is to obtain the maximum horizontal coordinate value of the lower half face occlusion area on the X axis, y.min and y.max are the minimum and maximum vertical coordinate values of the lower half face occlusion area on the Y axis respectively. Among them, Figure 3 The area enclosed by the rectangular frame in is the aforementioned lower half face occluded area, and the nose tip refers to the center point of the nose tip.
[0055] For the above step S300, based on the lower half face occlusion prompt text and the diffusion model, the foreground target is fused to the lower half face occlusion area of the background image B, which includes the following steps S310 and S330.
[0056] Step S310: Obtaining the encoded image of the foreground target image and the encoded image of the background image
[0057] The diffusion model is used to encode the foreground target image F and the background image B to obtain the encoded image and the encoded image In the above encoding process, the diffusion model is used to start from the image x at time 0 0 x at step length t t The solution process can be expressed as:
[0058] Wherein, P is text information. For example, when the foreground object is a hat, P can be expressed as the hat blocking the lower half of the face. The area range of the lower half of the face can be obtained in step S200. The text information in the input diffusion model can be set in advance. When the text information "hat blocking the lower half of the face" is described in English in the input model, O represents the space between characters, that is, the space between English words. t represents the scaling factor at the step time, where ε(x t ,t,P,O) is the noise fitted by the diffusion model.
[0059] Step S320: Encode the image and the encoded image Fusion to obtain the foreground and background coded images
[0060]
[0061] Where z is Gaussian noise, and z is the noise image at step t derived from the diffusion process of the diffusion model in S310. The image area corresponding to the lower half of the face occlusion area of the background image B is called the main background area b, X b The mask image of the main background area and the pixel value of the main background area in the mask image is 1, X F is the mask image corresponding to the foreground image fused on the occluded area of the lower half of the face, and the pixel value of the foreground target area in the mask image is 1. is the XOR result, and ⊙ is the XOR operation.
[0062] In the above steps, The XOR result of the main background area and the foreground target area is used to better connect the edge transition area. The combination of z repairs the transition area by utilizing the prior knowledge based on text prompt information of the text-driven diffusion model, which enhances X b and X F The smoothness of the transition between regions inside and outside the mask image.
[0063] Step S330: Using the text-driven diffusion model to encode the foreground and background images based on the lower face occlusion hint De-noising and decoding are performed to obtain the fused image S with the lower half of the face occluded.
[0064] In step S300, the foreground and background coded images Equivalent to the image to be denoised, step S300 uses the prompt text to drive the diffusion model to perform an inversion denoising process to obtain the final fused image S without noise.
[0065] In step S300, the following is specifically embodied: the background coded image The lower half face occlusion prompt text information and foreground target information are input into the denoising network superimposed with the attention module of the diffusion model for iterative denoising to obtain the fused image S of the lower half face occlusion after denoising.
[0066] Among them, the lower half face occlusion prompt text information includes the face area in the background image B and the lower half face occlusion area of the background image B indicated by the prompt text information, and the foreground target information is the extracted mask image of the foreground target.
[0067] Furthermore, the diffusion model is based on the DDPM model, and the denoising network is based on the UNET network; the attention map obtained by inputting the attention module is:
[0068]
[0069]
[0070] Among them, t represents the time step of the denoising solution of the diffusion model to obtain the fused image S, and l represents the lth layer of the UNET network; and are respectively the spatial features of the foreground target image F and the background area B extracted by the network at layer l when the time step is t, and the background area B is the occluded area of the lower half of the face; and are respectively the query vector and key vector of the lth layer extracted when the attention module of the UNET network processes the background area B at time step t, and are the query vector and key vector of the lth layer extracted when the attention module of the UNET network processes the foreground target image F at time step t, and Respectively, vector and The transpose of d 1 and d 2 They are and The dimension of the vector in ; and They are the self-attention map about the background image, the self-attention map about the foreground target image, and the self-attention map about and The attention diagram between the crosses; Soft max(.) indicates the normalization operation; V compose Constructing composite self-attention map based on regional pixel index The function factor of . Among them, Figure 4 for Figure 2 The foreground object in Figure 3 Schematic diagram of the fused image S formed by occluding the lower half of the face in FIG.
[0071] Further, and is the value vector of the lth layer extracted when the attention module of the UNET network processes the foreground target image F at time step t, is the value vector of the lth layer extracted when the attention module of the UNET network processes the background area B at time step t; and Respectively learnable projection matrices.
[0072] In the face occlusion fusion method based on text control, the occlusion data of the lower half of the face is automatically synthesized based on text control, and the image-guided synthesis of the diffusion model is realized by driving the text. The intermediate synthesis process is mainly controlled based on the text prompt, which is convenient for connecting and transitioning the overall style and edges of the foreground and background images to ensure their appearance consistency. Compared with traditional algorithms, the effect and speed are far better than the foreground and background harmonization (adjusting brightness, contrast, color saturation, etc.) of traditional images. The image inversion and synthesis are played to the best effect to assist in the synthesis of the occlusion data of the lower half of the face. At the same time, it also supports cross-domain foreground and background data synthesis in different style fields, which can provide a rich data source for the model and give full play to the advantages of data synthesis.
[0073] The face fusion system provided in the embodiment of the present application includes an image acquisition unit and an image processor connected to the image acquisition unit. The image acquisition unit is used to acquire a foreground image and a background image, and the image processor is used to execute the steps of the above-mentioned face fusion method based on text control, that is, the image processor is used to execute the specific steps of the above-mentioned steps S100, S200, and S300.
[0074] The computer-readable medium provided in the embodiment of the present application stores program code, and when the computer program code runs on a computer, the computer executes the steps of the face fusion method based on text control.
[0075] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A face occlusion fusion method based on text control, characterized in that: include: Obtaining a foreground target image for occlusion; Obtain the lower half face occlusion area of the background image B containing the face; as well as Based on the lower face occlusion hint text and the diffusion model, the foreground target is fused to the lower face occlusion area of the background image B.
2. The face occlusion fusion method based on text control as claimed in claim 1, characterized in that: The step of obtaining the lower face occlusion area of the background image B containing the face includes: Perform facial key point detection on the background image B to obtain the coordinates of three key points of the face: the center point of the left eye area (left.x, left.y), the center point of the right eye area (right.x, right.y) and the coordinates of the nose tip (nose.x, nose.y); The lower face occlusion area is located according to the coordinates of the three key points: the coordinate range of the lower face occlusion area on the X-axis is (0, x.up) and the coordinate range on the Y-axis is (y.min, y.max); Among them, left.x, left.y, right.x and right.y are the horizontal coordinates of the center point of the left eye area, the vertical coordinates of the center point of the left eye area, the horizontal coordinates of the center point of the right eye area, and the vertical coordinates of the center point of the right eye area respectively; y.min=min(right.y,left.y), y.max=y.min+(nose.yy.min)*2 / 5; x.up=|left.x-center.x|, center.x=(right.x-left.x) / 2; |.| means finding the absolute value, min(.) means taking the minimum value of the two, x.up is to obtain the maximum horizontal coordinate value of the lower half of the face occluded area on the X-axis, y.min and y.max are the minimum and maximum vertical coordinate values of the lower half of the face occluded area on the Y-axis respectively.
3. The face occlusion fusion method based on text control as claimed in claim 1, characterized in that: Based on the lower face occlusion hint text and the diffusion model, the foreground target is fused to the lower face occlusion area of the background image B, including: Get the encoded image of the foreground target image and the encoded image of the background image Encoded image and the encoded image Fusion to obtain the foreground and background coded images Text-driven diffusion model for encoding foreground and background images based on lower face occlusion hints De-noising and decoding are performed to obtain the fused image S with the lower half of the face occluded.
4. The face occlusion fusion method based on text control as claimed in claim 3, characterized in that: Background encoding image based on text-driven diffusion model with lower face occlusion hint De-noising and decoding are performed to obtain the fused image S with the lower half of the face blocked, which is: Encode the background image The lower half face occlusion prompt text information and foreground target information are input into the denoising network superimposed with the attention module of the diffusion model for iterative denoising to obtain the fused image S of the lower half face occlusion after denoising. Among them, the lower half face occlusion prompt text information includes the face area in the background image B and the lower half face occlusion area of the background image B indicated by the prompt text information, and the foreground target information is the extracted mask map of the foreground target.
5. The face occlusion fusion method based on text control as claimed in claim 5, characterized in that: The diffusion model is based on the DDPM model, and the denoising network is based on the UNET network; the attention map obtained by inputting the attention module is: Among them, t represents the time step of the denoising solution of the diffusion model to obtain the fused image S, and l represents the lth layer of the UNET network; and are respectively the spatial features of the foreground target image F and the background area B extracted by the network at layer l when the time step is t, and the background area B is the occluded area of the lower half of the face; and are respectively the query vector and key vector of the lth layer extracted when the attention module of the UNET network processes the background area B at time step t, and are the query vector and key vector of the lth layer extracted when the attention module of the UNET network processes the foreground target image F at time step t, and Respectively, vector and is the transpose of; d1 and d2 are and The vector dimensions in and They are the self-attention map about the background image, the self-attention map about the foreground target image, and the self-attention map about and The attention diagram between the crosses; Soft max(.) indicates the normalization operation; V compose Constructing composite self-attention map based on regional pixel index The function factor of .
6. The face occlusion fusion method based on text control as claimed in claim 3, characterized in that: Encoded image and the encoded image Fusion to obtain the foreground and background coded images for: Among them, z is Gaussian noise, the image area corresponding to the lower half of the face occlusion area of the background image B is called the main background area b, and X b is the mask image of the main background area, X F is the mask image corresponding to the foreground image fused to the occluded area of the lower half of the face, is the XOR result, and ⊙ is the XOR operation.
7. A face fusion system, characterized in that: include: An image acquisition unit, acquiring a foreground image and a background image; as well as An image processor is connected to the image acquisition unit and is used to execute the steps of the face occlusion and fusion method based on text control as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: The computer-readable medium stores program codes, and when the computer program codes are executed on a computer, the computer executes the steps of the face occlusion and fusion method based on text control as claimed in any one of claims 1 to 6.