An intelligent design system for face replacement
Through the face detection, feature extraction and image fusion module of the intelligent design system, the automation and complex feature analysis problems of face processing in the video picture are solved, the precise positioning and natural replacement of faces in the video stream are realized, and the intelligent video is improved Application processing capability.
Patent Information
- Application Number
- CN202311680842.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-08
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-08
AI Technical Summary
The existing technology lacks technical means to automatically and accurately locate, express, analyze and process the entire process of faces in video images, resulting in poor face detection and recognition effect in video streaming scenes, insufficient expression and understanding of complex facial features, and lack of adaptability. A systematic solution for face fusion.
An intelligent design system was designed, including a face detection and tracking module, a face feature extraction module, a face matching evaluation module and an adaptive image fusion module, and a pre-trained MTCNN face detection model and key points to detect convolutional nerves. Network and deep learning feature extraction, face replacement is performed through facial area feature alignment and matching evaluation, and image fusion algorithm is used.
It realizes fully automated detection, recognition, positioning, matching analysis and adaptive fusion replacement of faces in video images, improves the accuracy and nature of face processing, supports a variety of intelligent video applications, and ensures complex facial features and Adaptive expression and fusion of textures.
Smart Images

Figure CN118052723B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and more specifically, to an intelligent design system for face replacement. Background Art
[0002] The patent with application publication number CN116403260A discloses an AI face replacement method, including the following steps: S1: establishment of a face library; S2: face extraction; S3: model training; S4: face conversion; the S1 specifically includes: S11: establishment of a face generation library; S12: establishment of a face replacement library; the S2 specifically includes: S21: face detection; S22: face key point detection and alignment; S23: face segmentation; the S3 specifically includes: S31: generalized generation model training; S32: specific face model training; the S4 specifically includes: S41: face exchange; S42: face fusion. The AI face replacement method of the present invention can eliminate irregular occluders, such as hair, hands, glasses, etc., contained in the face image obtained in the face detection step through face segmentation, so as to ensure that the model can converge stably during the training process, will not regard the occluders as face information, and ensure that the final face-changing effect is more seamless and smooth, thereby improving the accuracy and stability of the replaced face.
[0003] The problem with the existing technology is that there is a lack of technical means to achieve automatic and precise positioning, expression, analysis and processing of faces in video images throughout the entire process; the main challenges include: it is difficult to ensure the effect of face detection and recognition in video streaming scenarios; the expression and understanding of complex facial features are also relatively weak; there is no decision-making ability to adaptively analyze and process facial areas; there is a lack of systematic solutions that organically integrate face positioning, analysis and processing for adaptive face fusion; the existence of these problems restricts the intelligent processing effect of facial areas in current video editing applications.
[0004] In view of this, the present invention proposes an intelligent design system for face replacement to solve the above problems. Summary of the invention
[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: an intelligent design system for face replacement, comprising:
[0006] The face detection and tracking module is used to detect faces on the input video stream and obtain the face area image;
[0007] A face feature extraction module is used to extract a facial key point coordinate sequence and deep learning features from a face region image, and obtain facial region features based on the facial key point coordinate sequence and deep learning features;
[0008] A face matching evaluation module is used to compare face region features between the face region image to be replaced and the target face image, divide the face region image to be replaced and the target face image into n sub-regions, and obtain n replacement matching degrees between the face region image to be replaced and the n sub-regions of the target face image;
[0009] The adaptive image fusion module sets a matching degree threshold interval and obtains the matching result by comparing n replacement matching degrees and the matching degree threshold interval; the image fusion algorithm is used according to the comparison result to fuse and replace the target portrait image and the image of the face area to be replaced; the modules are connected by wired and / or wireless means to realize data transmission between modules.
[0010] Furthermore, the method of performing face detection on the input video stream to obtain the face area image includes:
[0011] The pre-trained MTCNN face detection model is loaded into the memory; the input video stream is read frame by frame to extract the RGB image; the extracted RGB image is input into the pre-trained MTCNN face detection model to generate the coordinate frame of each face; according to the face coordinate frame, the area where the face frame is located is intercepted in the input RGB image to generate the face area image;
[0012] The pre-training process of the MTCNN face detection model includes:
[0013] The training of the MTCNN face detection model is divided into three stages, including training the extraction network, training the optimization network, and training the output network.
[0014] Furthermore, the training extraction network includes:
[0015] Collect m images including human faces as a training set, where m is an integer greater than 1; perform manual or computer face annotation on the training set images to obtain an image training set including a face label frame;
[0016] The extraction network adopts a convolutional neural network structure, which includes a convolutional layer, an activation function layer, a pooling layer, and a fully connected layer;
[0017] Set the input layer to the three-channel pixel data of the input RGB image; use b convolution kernels to perform convolution operation on the input RGB image to obtain the feature map; use the activation function layer to apply nonlinear activation to the feature map output by the convolution layer; use the pooling layer to reduce the dimension of the feature map; obtain the reduced dimension feature map; use the fully connected layer to convert the reduced dimension feature map into a feature map vector;
[0018] The loss function layer uses the cross entropy function H(P,Q)=-∑ i P(i)log(Q(i));
[0019] Among them, H(P,Q) represents the cross entropy loss between the true distribution P and the model predicted distribution Q; P(i) represents the probability of the i-th category in the true distribution; Q(i) represents the probability of the i-th category in the model predicted distribution;
[0020] Define the regression box loss function to measure the error between the network prediction box and the label box;
[0021] The regression box loss function is:
[0022]
[0023]
[0024] The value of E is p x -g x , p y -g y , p w -g w and p h -g h ;
[0025] Among them, (p x , p y ) is the center coordinate of the prediction box, p w is the width of the prediction box, p h is the height of the prediction box; (g x , g y ) is the center coordinate of the label box, g w is the width of the label box, g h is the height of the label box; N is the number of images in the total image training set;
[0026] The back propagation algorithm is used to update the network parameters according to the loss function, so that the prediction box continues to approach the label box; by iteratively optimizing the model, the extraction network is finally obtained.
[0027] Furthermore, the steps of training and optimizing the network include:
[0028] Use the extraction network to evaluate all candidate boxes generated by the training set images; evaluate the IoU between each candidate box and the label box; IoU indicates the degree of overlap between the two;
[0029] If the IoU of the candidate box is greater than 0.5, the candidate box is marked as a positive sample; if the IoU of the candidate box is less than or equal to 0.5, the candidate box is marked as a negative sample;
[0030] Define the network structure of the optimization network, which includes convolutional layers, pooling layers, and fully connected layers; define the training goal of the optimization network as outputting a binary judgment result, which is whether the current candidate box includes a face; expect the judgment result to be consistent with the annotated label; the label includes positive samples and negative samples;
[0031] The binary cross entropy loss function is used as the loss function of the optimization network;
[0032] Binary cross entropy loss function Loss(o,y)=-(ylog(o)+(1-y)log(1-o));
[0033] Among them, o is the probability that the current sample is predicted to be a positive sample; y is the label of the current sample, which is 1 for positive samples and 0 for negative samples;
[0034] Input the image with the candidate box to the optimized network for network calculation to obtain the binary judgment result; compare the binary judgment result with the label and calculate the value of the binary cross entropy loss function; update the network parameters of the optimized network through back propagation and other algorithms. The value of the binary cross entropy loss function will gradually decrease during the update and optimization process. When its decrease rate slows down, the training is terminated; and the optimized network is obtained.
[0035] Furthermore, the step of training the output network includes:
[0036] After being filtered by the optimized network, the remaining high-quality candidate frame images are used as training samples for the output network; the network structure for constructing the output network is the same as that of the optimized network;
[0037] The loss function of the output network is defined as a coordinate regression loss function, which is used to measure the sum of square errors between the output network prediction box and the label box in terms of position and size.
[0038] The input of the output network is the candidate box image, and the output of the output network is the adjusted prediction box coordinates; the prediction box coordinates include the center coordinates of the prediction box, the width of the prediction box, and the height of the prediction box;
[0039] Predefine the parameters of the output network, and the output network predicts the coordinates of the candidate box after the output adjustment according to the current parameters; calculate the coordinate regression loss function value of the predicted box and the image label box; calculate the gradient of each parameter to the coordinate regression loss function value by derivation by the chain rule; fine-tune the parameters of the output network along the negative gradient direction to reduce the value of the coordinate regression loss function; when the reduction speed slows down, end the training; and obtain the output network;
[0040] After the three stages are trained, they are connected in series to form a complete MTCNN face detection model.
[0041] Furthermore, the extraction process of the facial key point coordinate sequence includes:
[0042] Construct a key point detection convolutional neural network, which includes a convolutional layer, a pooling layer, and a fully connected layer; the input of the key point detection convolutional neural network is a face area image; the face area image is an RGB image; the output of the key point detection convolutional neural network is the coordinates of the face key points, and there are c face key points in total;
[0043] Collect and annotate v facial image feature images to form a facial image feature image set;
[0044] The collected facial image feature images cover different genders, ages, races and facial expressions and postures; the coordinates of c key points of the face in each facial image feature image are manually or computer marked using an image annotation tool;
[0045] The labeled facial image feature set is input into the key point detection convolutional neural network; the loss function is the Euclidean distance between the predicted key point coordinates and the labeled key point coordinates;
[0046] The loss function on a single face image feature image is
[0047] in, The jth key point coordinate of the i-th image predicted by the key point detection convolutional neural network; y ij The coordinates of the jth key point manually annotated for the i-th image; To predict the key point coordinates And the coordinate y of the key point ij The Euclidean distance of ; c is the number of facial key points;
[0048] On the face image feature image set, the overall loss function of the key point detection model is:
[0049]
[0050] Among them, v is the number of all images included in the face image feature image set;
[0051] Define the network parameters of the key point detection convolutional neural network and initialize it; input images in the face image feature image set for training, and forward propagate to calculate the loss L z ; Calculate the loss L through the back propagation algorithm z The gradient of the parameter θ By gradient descent Among them, α is the learning rate; through iterative optimization, the loss L is gradually reduced. z The value of , get the face key point detection model;
[0052] Input the face region image into the face key point detection model to predict and extract the output c key point coordinates; construct the face key point coordinate sequence from the c key point coordinates;
[0053] The deep learning feature extraction process includes:
[0054] Predefine the middle layer of the face key point detection model; input the face area image into the face key point detection model; extract the deep feature vector output by the middle layer of the face key point detection model; the deep feature vector is the deep learning feature;
[0055] The method of acquiring the face region feature includes:
[0056] The depth feature vector is a fixed-length vector, and the facial key point coordinate sequence is of variable length; the facial key point coordinate sequence is flattened, and each key point is represented by a vector of length 2; the depth feature vector is concatenated with the flattened facial key point coordinate sequence to obtain a fixed-length vector.
[0057] Furthermore, the method for obtaining the replacement matching degree includes:
[0058] The facial feature extraction module is used to extract the facial region features of the face region image A to be replaced and the facial region features of the target face image B respectively; n pairs of facial region features are divided according to n sub-regions; the facial region features are high-dimensional vectors, including two parts of information: deep learning features and facial key point coordinate sequences; the cosine similarity sim(a i ,b i );
[0059]
[0060] Among them, a i is the facial region feature of the face region image A to be replaced in the i-th sub-region; b i is the face region feature of the target face image B in the i-th sub-region; a i .b i Indicates a i and b i Inner product of vectors; ||a i || and ||b i || respectively represent a i and b i The L2 norm of ;
[0061] sim(a i ,b i ) is used as the replacement matching degree between the face area image to be replaced and the target face image sub-area.
[0062] Furthermore, the matching degree threshold interval is set in the following manner:
[0063] Collect a training data set including u pairs of face images, each pair of face images is a face image of the same person taken in different scenes; calculate the matching score for each pair of face images ij ; Count the matching scores of all face images in the entire training data set ij , calculate the matching score of all face images ij The mean Calculate the matching score of all face images ij Standard Deviation Where u is the number of face image pairs in the training dataset;
[0064] Set the upper limit of the matching threshold interval to ρ+3σ; set the lower limit of the matching threshold interval to ρ-3σ; then the matching threshold interval is [ρ+3σ,ρ-3σ];
[0065] The comparison result is a high match, a medium match, or a low match;
[0066] If the replacement matching degree among the n replacement matching degrees is greater than or equal to, the comparison result of the corresponding sub-region is a high match; if the replacement matching degree among the n replacement matching degrees is within the matching degree threshold interval, the comparison result of the corresponding sub-region is a medium match; if the replacement matching degree among the n replacement matching degrees is less than or equal to ρ-3σ, the comparison result of the corresponding sub-region is a low match.
[0067] Furthermore, the method of using an image fusion algorithm according to the comparison result to fuse and replace the target portrait image and the face area image to be replaced includes:
[0068] The target portrait image with a high match in the comparison result and the sub-region of the face region to be replaced are fused and replaced using the Poisson fusion algorithm;
[0069] The target portrait image and the sub-region of the face area to be replaced are fused and replaced by using the color migration algorithm;
[0070] The target portrait image with a low match in comparison result and the sub-region of the face area image to be replaced are fused and replaced using a geometric transformation algorithm; and the fusion replacement between the target portrait image and the sub-region of the face area image to be replaced is achieved.
[0071] An intelligent design method for face replacement, which is implemented based on the intelligent design system for face replacement, comprises: S1, performing face detection on an input video stream to obtain a face area image;
[0072] S2, extracting facial key point coordinate sequence and deep learning features from the facial region image, and obtaining facial region features according to the facial key point coordinate sequence and the deep learning features;
[0073] S3, comparing facial region features of the facial region image to be replaced with the target facial image, dividing the facial region image to be replaced and the target facial image into n sub-regions, and obtaining n replacement matching degrees between the facial region image to be replaced and the n sub-regions of the target facial image;
[0074] S4, setting a matching degree threshold interval, and obtaining a matching result by comparing n replacement matching degrees and the matching degree threshold interval; and using an image fusion algorithm according to the matching result to fuse and replace the target portrait image and the face area image to be replaced.
[0075] Technical effects and advantages of an intelligent design system for face replacement according to the present invention:
[0076] It realizes fully automated processing of detection, recognition, positioning, matching analysis and adaptive fusion and replacement of faces in video images; the system integrates face detection, feature expression, similarity judgment and image fusion modules, which can accurately identify, express and analyze face areas, and select the optimal algorithm according to the judgment results to realize natural fusion and replacement of faces; in this way, while ensuring the processing effect, it also realizes the automatic selection of processing strategies and flexible expansion of the system; overall, this system has the face processing capability to support a variety of intelligent video applications, which not only ensures the ability to accurately locate face areas, but also realizes the adaptive expression, analysis and fusion of complex facial features and textures. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] Figure 1 A schematic diagram of an intelligent design system for face replacement according to the present invention;
[0078] Figure 2 A schematic diagram of key points of a human face according to the present invention;
[0079] Figure 3 A schematic diagram of an intelligent design method for face replacement according to the present invention;
[0080] Figure 4 It is a schematic diagram of an electronic device of the present invention. DETAILED DESCRIPTION
[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0082] Example 1
[0083] See also Figure 1 As shown, the intelligent design system for face replacement described in this embodiment includes: a face detection and tracking module, which is used to perform face detection on an input video stream and obtain a face area image;
[0084] A face feature extraction module is used to extract a facial key point coordinate sequence and deep learning features from a face region image, and obtain facial region features based on the facial key point coordinate sequence and deep learning features;
[0085] A face matching evaluation module is used to compare face region features between the face region image to be replaced and the target face image, divide the face region image to be replaced and the target face image into n sub-regions, and obtain n replacement matching degrees between the face region image to be replaced and the n sub-regions of the target face image;
[0086] The adaptive image fusion module sets a matching degree threshold interval, obtains a matching result by comparing n replacement matching degrees and the matching degree threshold interval; uses an image fusion algorithm according to the matching result to fuse and replace the target portrait image and the face area image to be replaced; and connects the modules by wired and / or wireless means to realize data transmission between the modules;
[0087] Furthermore, the method of performing face detection on the input video stream to obtain the face area image includes:
[0088] The pre-trained MTCNN face detection model is loaded into the memory; the input video stream is read frame by frame to extract the RGB image; the extracted RGB image is input into the pre-trained MTCNN face detection model to generate the coordinate frame of each face; according to the face coordinate frame, the area where the face frame is located is intercepted in the input RGB image to generate the face area image;
[0089] The pre-training process of the MTCNN face detection model includes:
[0090] The training of the MTCNN face detection model is divided into three stages, including training the extraction network, training the optimization network, and training the output network.
[0091] The method of training the extraction network includes:
[0092] Collect m images including human faces as a training set, where m is an integer greater than 1; perform manual or computer face annotation on the training set images to obtain an image training set including a face label frame;
[0093] The extraction network adopts a convolutional neural network structure, which includes a convolutional layer, an activation function layer, a pooling layer, and a fully connected layer;
[0094] Set the input layer to the three-channel pixel data of the input RGB image; use b convolution kernels to perform convolution operation on the input RGB image to obtain the feature map; use the activation function layer, such as the ReLU layer, to apply nonlinear activation to the feature map output by the convolution layer; use the pooling layer to reduce the dimension of the feature map; obtain the reduced-dimensional feature map; use the fully connected layer to convert the reduced-dimensional feature map into a feature map vector;
[0095] The loss function layer, for example, mean squared error and cross entropy, measures the gap between the predicted output of the extraction network and the true label;
[0096] In a preferred embodiment, the loss function layer uses cross entropy;
[0097] Cross entropy H(P,Q) = -∑ i P(i)log(Q(i));
[0098] Among them, H(P,Q) represents the cross entropy loss between the true distribution P and the model predicted distribution Q; P(i) represents the probability of the i-th category in the true distribution. For the face extraction task, it can be regarded as a label of whether a face is contained; Q(i) represents the probability of the i-th category in the model predicted distribution;
[0099] Define the regression box loss function to measure the error between the network prediction box and the label box;
[0100] The regression box loss function is:
[0101]
[0102]
[0103] The value of E is p x -g x , p y -g y , p w -g w and p h -g h ;
[0104] Among them, (p x , p y ) is the center coordinate of the prediction box, p w is the width of the prediction box, p h is the height of the prediction box; (g x , g y ) is the center coordinate of the label box, g w is the width of the label box, g his the height of the label box; N is the number of images in the total image training set;
[0105] The back propagation algorithm is used to update the network parameters according to the loss function, so that the prediction box is constantly close to the label box; the extraction network is finally obtained by iteratively optimizing the model;
[0106] It should be noted that in the parameter setting of the extraction network, the number and size of convolution kernels, the choice of activation function, the pooling method, etc. will affect the final extraction effect; the best parameter combination is obtained through multiple adjustments and tests;
[0107] The steps of training and optimizing the network include:
[0108] Use the extraction network to evaluate all candidate boxes generated by the training set images; evaluate the IoU between each candidate box and the label box; IoU indicates the degree of overlap between the two;
[0109] If the IoU of the candidate box is greater than 0.5, the candidate box basically contains the face accurately, and the candidate box is marked as a positive sample; if the IoU of the candidate box is less than or equal to 0.5, the candidate box and the label box basically do not overlap, and the face cannot be correctly detected, and the candidate box is marked as a negative sample;
[0110] Define the network structure of the optimization network, which includes convolutional layers, pooling layers, and fully connected layers; define the training goal of the optimization network as outputting a binary judgment result, which is whether the current candidate box includes a face; expect the judgment result to be consistent with the annotated label; the label includes positive samples and negative samples;
[0111] The binary cross entropy loss function is used as the loss function of the optimization network;
[0112] Binary cross entropy loss function Loss(o,y)=-(ylog(o)+(1-y)log(1-o));
[0113] Among them, o is the probability that the current sample is predicted to be a positive sample; y is the label of the current sample, which is 1 for positive samples and 0 for negative samples;
[0114] Input the image with the candidate box to the optimized network for network calculation to obtain the binary judgment result; compare the binary judgment result with the label and calculate the binary cross entropy loss function value; update the network parameters of the optimized network through back propagation and other algorithms to reduce the binary cross entropy loss function value and improve the judgment accuracy;
[0115] As the iteration proceeds, the value of the binary cross entropy loss function will gradually decrease. When the rate of decrease slows down, the optimized network tends to be saturated, and the training ends. The optimized network is obtained.
[0116] When the candidate boxes are input into the model, most negative samples will be filtered out, thus improving the accuracy of subsequent detection;
[0117] The step of training the output network comprises:
[0118] After being filtered by the optimized network, the remaining high-quality candidate box images are used as training samples for the output network;
[0119] The network structure of the output network is the same as that of the optimized network;
[0120] The loss function of the output network is defined as a coordinate regression loss function, which is used to measure the sum of square errors between the output network prediction box and the label box in terms of position and size.
[0121] It needs to be explained that the coordinate regression loss function is calculated as:
[0122] 1. Calculate the difference between the horizontal coordinate px of the center of the prediction box and the horizontal coordinate gx of the center of the label box, and square it;
[0123] 2. Calculate the difference between the vertical coordinate py of the center of the prediction box and the vertical coordinate gy of the center of the label box, and square it;
[0124] 3. Calculate the difference between the prediction box width pw and the label box width gw and square it;
[0125] 4. Calculate the difference between the predicted box height ph and the label box height gh, and square it;
[0126] The horizontal and vertical coordinates are based on the image coordinate system, and the coordinate point is the left side of the pixel point in the image coordinate system;
[0127] The sum of the squares of items 1-4 above is the final value of the coordinate regression loss function;
[0128] The input of the output network is the candidate box image, and the output of the output network is the adjusted prediction box coordinates; the prediction box coordinates include the center coordinates of the prediction box, the width of the prediction box, and the height of the prediction box;
[0129] Predefine the parameters of the output network, and the output network predicts the coordinates of the candidate box after the output adjustment according to the current parameters; calculate the coordinate regression loss function value of the predicted box and the image label box; calculate the gradient of each parameter to the coordinate regression loss function value by derivation by the chain rule; fine-tune the parameters of the output network along the negative gradient direction to reduce the value of the coordinate regression loss function; when the reduction speed slows down, end the training; and obtain the output network;
[0130] Adjust the output network parameters to gradually reduce the position and size difference between the prediction box and the label box, making the coordinate prediction more accurate
[0131] After all three stages are trained, the three stages are used in series to form a complete MTCNN face detection model;
[0132] Furthermore, the extraction process of the facial key point coordinate sequence includes:
[0133] Construct a key point detection convolutional neural network, which includes a convolutional layer, a pooling layer, and a fully connected layer;
[0134] The input of the key point detection convolutional neural network is the face area image; the face area image is an RGB image;
[0135] The output of the key point detection convolutional neural network is the coordinates of the key points of the face. The key points of the face include the corners of the eyes, the center of the eyebrows, the corners of the mouth, etc., a total of c key points; Figure 2 As shown,
[0136] Taking 68 key points as an example, there are 17 contour feature points, including 8 points for the left and right eye corners and eyebrow corners; 4 points for the bridge of the nose, 1 point for each of the left and right nose wings; 2 points for the upper lip and 2 points for the lower jaw;
[0137] There are 22 feature points of the eyebrows, including 11 points for each eyebrow, which describe the outline and details of the eyebrow shape; there are 20 feature points of the eyes, including 10 points for each eye, including the outline of the upper and lower eyelids and the points of the eyeball; there are 9 feature points of the nose, including 1 point at the tip of the nose, 4 points on both sides of the nose, and 4 points on the midline of the nose base;
[0138] Collect and annotate v facial image feature images to form a facial image feature image set;
[0139] The collected facial image feature images cover different genders, ages, races and facial expressions and postures; the coordinates of c key points of the face in each facial image feature image are manually or computer marked using an image annotation tool;
[0140] The labeled facial image feature set is input into the key point detection convolutional neural network; the loss function is the Euclidean distance between the predicted key point coordinates and the labeled key point coordinates;
[0141] The loss function on a single face image feature image is
[0142] in, The jth key point coordinate of the i-th image predicted by the key point detection convolutional neural network; y ij The coordinates of the jth key point manually annotated for the i-th image; To predict the key point coordinates And the coordinate y of the key point ijThe Euclidean distance of ; c is the number of facial key points;
[0143] On the face image feature image set, the overall loss function of the key point detection model is:
[0144]
[0145] Among them, v is the number of all images included in the face image feature image set;
[0146] Define the network parameters and initialization parameters of the key point detection convolutional neural network;
[0147] Input the images in the face image feature set for training, and forward propagate to calculate the loss L z ; Calculate the loss L through the back propagation algorithm z The gradient of the parameter θ By gradient descent Among them, α is the learning rate;
[0148] Through iterative optimization, the loss L is gradually reduced. z The value of , get the face key point detection model;
[0149] Input the face region image into the face key point detection model to predict and extract the output c key point coordinates; construct the face key point coordinate sequence from the c key point coordinates;
[0150] Furthermore, the deep learning feature extraction process includes:
[0151] Predefine the middle layer of the face key point detection model; input the face area image into the face key point detection model; extract the deep feature vector output by the middle layer of the face key point detection model; the deep feature vector is the deep learning feature;
[0152] It should be noted that the middle layer is the middle convolution layer of the face key point detection model, which can also be called the feature extraction layer. For example, in the ResNet-50 model, the third convolution layer is selected as the feature extraction layer; a single face area image is input, and the model forward propagation calculation is started. The calculation process will pass through the defined feature extraction layer and output the recognition result; when in the feature extraction layer, its activation map is output, and the activation map retains the high-level semantic information of the input face image, that is, as a deep feature representation of the face image; the deep feature representation is post-processed, including adjusting the feature map size, L2 regularization normalization feature values to the same scale, and finally forming a fixed-length deep feature vector;
[0153] Furthermore, the method of acquiring the facial region features includes:
[0154] The depth feature vector is a fixed-length vector, such as 1024 dimensions; it records the global information of the image; the facial key point coordinate sequence is of variable length, such as c horizontal and vertical coordinate pairs; it mainly describes local details; the facial key point coordinate sequence is flattened, and each key point is represented by a vector of length 2;
[0155] Directly concatenate the depth feature vector with the flattened facial key point coordinate sequence to obtain a fixed-length vector that includes both the global information of the depth feature and the local information of the key points.
[0156] Furthermore, the method for obtaining the replacement matching degree includes:
[0157] The facial region features of the face region image A to be replaced and the face region features of the target face image B are respectively extracted by using the facial feature extraction module; n pairs of facial region features are divided according to n sub-regions;
[0158] The face region feature is a high-dimensional vector, which includes two parts of information: deep learning features and facial key point coordinate sequence;
[0159] Calculate the cosine similarity sim(a) between n pairs of face region features i ,b i );
[0160]
[0161] Among them, a i is the facial region feature of the face region image A to be replaced in the i-th sub-region; b i is the face region feature of the target face image B in the i-th sub-region; a i .b i Indicates a i and b i Inner product of vectors; ||a i || and ||b i || respectively represent a i and b i The L2 norm of ;
[0162] It should be noted that sim(a i ,b i ) has a value range of [-1,1]; when a i and b i When the two vectors are in exactly the same direction, the value is 1; when the vectors are perpendicular, the value is 0; when a i and b i When the vector directions are completely opposite, the value is -1;
[0163] According to sim(a i ,b i) value to determine the difference in the features of the two faces; the larger the value, the more similar they are, and the value close to 1 indicates a high match; sim(a i ,b i ) as the matching degree between the face area image to be replaced and the target face image;
[0164] It should be noted that the calculation method of the L2 norm is to take the square root of the sum of the squares of each element in the vector;
[0165] Furthermore, the matching degree threshold interval is set in the following manner:
[0166] Collect a training data set including u pairs of face images, each pair of face images is a face image of the same person taken in different scenes; calculate the matching score for each pair of face images ij ; Count the matching scores of all face images in the entire training data set ij , calculate the matching score of all face images ij The mean Calculate the matching score of all face images ij Standard Deviation Where u is the number of face image pairs in the training dataset;
[0167] According to Gaussian distribution theory, samples greater than or equal to ρ+3σ can be regarded as high matching samples, and the upper limit of the matching threshold interval is set to ρ+3σ; samples less than or equal to ρ-3σ can be regarded as low matching samples, and the lower limit of the matching threshold interval is set to ρ-3σ; then the matching threshold interval is [ρ+3σ,ρ-3σ];
[0168] Further, the comparison result is a high match, a medium match, or a low match;
[0169] If the replacement matching degree among the n replacement matching degrees is greater than or equal to , the comparison result of the corresponding sub-region is a high match; if the replacement matching degree among the n replacement matching degrees is within the matching degree threshold interval, the comparison result of the corresponding sub-region is a medium match; if the replacement matching degree among the n replacement matching degrees is less than or equal to ρ-3σ, the comparison result of the corresponding sub-region is a low match;
[0170] Furthermore, the method of using an image fusion algorithm according to the comparison result to fuse and replace the target portrait image and the face area image to be replaced includes:
[0171] The target portrait image with a high match in the comparison result and the sub-region of the face region to be replaced are fused and replaced using the Poisson fusion algorithm;
[0172] The target portrait image and the sub-region of the face area to be replaced are fused and replaced by using the color migration algorithm;
[0173] The target portrait image with a low match in the comparison result and the sub-region of the face area image to be replaced are fused and replaced by using a geometric transformation algorithm; the fusion replacement between the target portrait image and the face area image to be replaced is realized;
[0174] The process of fusion replacement by the Poisson fusion algorithm includes:
[0175] Step 101, extracting high matching sub-regions corresponding to the face image to be replaced A and the target face image B, and marking them as image I and image J respectively;
[0176] Step 102: Calculate the gradient direction field of the image I and the image J at each pixel position, obtain the guide field BI and the guide field, and represent the brightness change trend of the image I and the image J at each pixel point;
[0177] It should be noted that the process of calculating the gradient direction field is as follows:
[0178] Let GxI and GyI represent the gradient magnitude mapping of image I in the x direction and y direction respectively; GxI(x,y) and GxI(x,y) represent the horizontal and vertical gradient components at the position (x,y); the x and y directions are defined in the coordinate system of the image, where x and y are the horizontal and vertical coordinates of the pixel respectively;
[0179] Calculate the gradient magnitude
[0180] Calculate the gradient direction of the position
[0181] Construct the guidance field BI = (AmpI(x,y),θI(x,y));
[0182] The same processing is performed on the image J; that is, the guiding field BI and the guiding field BJ are obtained;
[0183] Step 103: Substitute the guiding field BI and the guiding field BJ into the Poisson equation to solve it, and obtain the density distribution mapping of the image I and the image J, which represents the pixel density of each pixel point, as the density map AI and the density map AJ;
[0184] The Poisson equation solution process includes:
[0185] Laplacian(g)=div(g0Δg0), where g represents the image pixel density distribution map to be obtained; g0 represents the given guidance field; Δ represents the Laplacian operator; div represents the divergence operation;
[0186] Let g be the density map AI, g0 be the guiding field BI; get Laplacian(AI)=div(BIΔBI);
[0187] Discretize Laplacian (AI) = div (BIΔBI) by finite difference, that is, obtain density map AI; perform the same processing on the guide field BJ, and obtain density map AJ;
[0188] Step 104, defining a fusion weight parameter β; the value of the fusion weight parameter β is defined to be between 0 and 1;
[0189] Calculate the new pixel density distribution Den = β × AI + (1-β) × AJ;
[0190] Step 105: According to the new pixel density distribution Den distribution mapping, the fusion contribution size of each pixel in the image I and the image J is controlled, and finally a new image after fusion and replacement is generated, and the Poisson fusion of the sub-images in the face area is completed;
[0191] The process of fusion replacement by the color migration algorithm includes:
[0192] Step 201, extract the matching sub-regions corresponding to the face image A to be replaced and the target face image B, and mark them as image Qx and image Qy respectively;
[0193] Step 202: Convert the RGB format image Qx and image Qy to the LAB color space to obtain image Qx1 and image Qy1;
[0194] Step 203, calculate the mean value mean I and the standard deviation dev I of the image Qx1 in the LAB color space; calculate the mean value meanJ and the standard deviation devJ of the image Qy1 in the LAB color space;
[0195] Step 204, assign the values of meanJ and devJ of image Qy1 to image Qx; that is, image Qx adopts the same color mean and standard deviation as image Qy; that is, the fusion replacement of the matching sub-regions in the face region is completed;
[0196] The process of fusion replacement by the geometric transformation algorithm includes:
[0197] Step 301, extracting low matching sub-region images corresponding to the face image to be replaced A and the target face image B, and marking them as sub-image src and sub-image tar respectively;
[0198] Step 302: extract the facial key point coordinate sequence rc detected in the sub-image src; extract the facial key point coordinate sequence ar detected in the sub-image tar;
[0199] Step 303: Calculate the geometric transformation matrix Me from the sub-image src and the sub-image tar based on the difference between the facial key point coordinate sequence rc and the facial key point coordinate sequence ar;
[0200] Step 304, applying the transformation matrix Me to remap the coordinates of each pixel in the sub-image src to obtain a new sub-image src', whose facial shape is consistent with the sub-image tar;
[0201] Step 305: fuse the new sub-image src' after geometric transformation with the pixel values in the corresponding sub-image tar, so as to complete the fusion replacement of the low matching sub-region of the face region;
[0202] Specifically, a function of constructing a geometric transformation matrix, the input of which is a facial key point coordinate sequence rc and a facial key point coordinate sequence ar; using the least squares method, the optimal affine transformation matrix Me corresponding to the mapping from the facial key point coordinate sequence rc to the facial key point coordinate sequence ar is calculated; the size of the matrix Me is 3x3, and the matrix Me includes the translation, rotation, and scaling information required for the image coordinate transformation; the output is the obtained geometric transformation matrix Me;
[0203] Traverse the coordinate position (xi, yi) of each pixel in the sub-image src; substitute (xi, yi) into the transformation matrix Me for matrix transformation calculation to obtain the coordinate mapping position (xi', yi');
[0204] Map the coordinates of all pixels to the positions and reconstruct them on the blank image to generate a sub-image src', whose facial shape is consistent with the sub-image tar; calculate the distance weight wr between the color RGB value of each pixel in the sub-image src and the new coordinates (xi', yi') in src';
[0205] According to the distance weight wr and the RGB color value of the corresponding position (xi', yi') in the sub-image tar, the fused pixel value RGB' of the position is calculated; through the above pixel fusion process, a new fused sub-image is generated on the sub-image src';
[0206] It should be noted that the process of obtaining the distance weight wr is as follows:
[0207] Calculate the Euclidean distance between the coordinate mapping position and the original coordinate position Define the Gaussian kernel function Among them, τ is the specified Gaussian kernel parameter; d is the independent variable that needs to participate in the operation;
[0208] The Gaussian kernel function is applied to the Euclidean distance DR as follows:
[0209] Then f(DR) is the distance weight wr.
[0210] This embodiment realizes fully automated processing of detection, recognition, positioning, matching analysis and adaptive fusion and replacement of faces in video images; the system integrates face detection, feature expression, similarity judgment and image fusion modules, can accurately recognize, express and analyze face areas, and select the optimal algorithm according to the judgment results to realize natural fusion and replacement of faces; in this way, while ensuring the processing effect, it also realizes the automated selection of processing strategies and the flexible expansion of the system; in general, the system has the face processing capability to support a variety of intelligent video applications, which not only ensures the ability to accurately locate face areas, but also realizes the adaptive expression, analysis and fusion of complex facial features and textures.
[0211] Example 2
[0212] See also Figure 3 As shown, the part not described in detail in this embodiment can be found in the description of embodiment 1, which provides an intelligent design method for face replacement, including: S1, performing face detection on an input video stream to obtain a face area image;
[0213] S2, extracting facial key point coordinate sequence and deep learning features from the facial region image, and obtaining facial region features according to the facial key point coordinate sequence and the deep learning features;
[0214] S3, comparing facial region features of the facial region image to be replaced with the target facial image, dividing the facial region image to be replaced and the target facial image into n sub-regions, and obtaining n replacement matching degrees between the facial region image to be replaced and the n sub-regions of the target facial image;
[0215] S4, setting a matching degree threshold interval, and obtaining a matching result by comparing n replacement matching degrees and the matching degree threshold interval; and using an image fusion algorithm according to the matching result to fuse and replace the target portrait image and the face area image to be replaced.
[0216] Example 3
[0217] See also Figure 4 As shown, according to another aspect of the present application, an electronic device is also provided. The electronic device may include an input device, an arithmetic unit, a controller, a main memory, and an output device. The main memory stores a computer readable code, and when the computer readable code is executed by one or more processors, the method for extracting production data of a foldable hollow board based on big data as described above can be executed.
[0218] The method or system according to the embodiment of the present application can also be used by Figure 4 The electronic device architecture shown in FIG. Figure 4As shown, the electronic device may include an input device, an arithmetic unit, a controller, a main memory, and an output device. A storage device in the electronic device, such as ROM503 or a hard disk, may store a method for extracting foldable hollow board production data based on big data provided in this application. Furthermore, the electronic device may also include a user interface. Of course, Figure 4 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 4 One or more components of an electronic device are shown.
[0219] In addition, according to the implementation of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided by the present application, an intelligent design method for face replacement. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.
Claims
1. An intelligent design system for face replacement, It is characterized in that include: The face detection and tracking module is used to detect faces on the input video stream and obtain the face area image; A face feature extraction module is used to extract a facial key point coordinate sequence and deep learning features from a face region image, and obtain facial region features based on the facial key point coordinate sequence and deep learning features; A face matching evaluation module is used to compare face region features between the face region image to be replaced and the target face image, divide the face region image to be replaced and the target face image into n sub-regions, and obtain n replacement matching degrees between the face region image to be replaced and the n sub-regions of the target face image; The adaptive image fusion module sets a matching degree threshold interval, obtains a matching result by comparing n replacement matching degrees and the matching degree threshold interval; uses an image fusion algorithm according to the matching result to fuse and replace the target portrait image and the face area image to be replaced; and connects the modules by wired and / or wireless means to realize data transmission between the modules; The method of performing face detection on the input video stream to obtain the face area image includes: The pre-trained MTCNN face detection model is loaded into the memory; the input video stream is read frame by frame to extract the RGB image; the extracted RGB image is input into the pre-trained MTCNN face detection model to generate the coordinate frame of each face; according to the face coordinate frame, the area where the face frame is located is intercepted in the input RGB image to generate the face area image; The pre-training process of the MTCNN face detection model includes: The training of the MTCNN face detection model is divided into three stages, including training the extraction network, training the optimization network, and training the output network. The method of training the extraction network includes: Collect m images including human faces as a training set, where m is an integer greater than 1; perform manual or computer face annotation on the training set images to obtain an image training set including a face label frame; The extraction network adopts a convolutional neural network structure, which includes a convolutional layer, an activation function layer, a pooling layer, and a fully connected layer; Set the input layer to the three-channel pixel data of the input RGB image; use b convolution kernels to perform convolution operation on the input RGB image to obtain the feature map; use the activation function layer to apply nonlinear activation to the feature map output by the convolution layer; use the pooling layer to reduce the dimension of the feature map; obtain the reduced dimension feature map; use the fully connected layer to convert the reduced dimension feature map into a feature map vector; The loss function layer uses the cross entropy function H(P,Q)=-∑ f P(f)log(Q(f)); Among them, H(P,Q) represents the cross entropy loss between the true distribution P and the model predicted distribution Q; P(f) represents the probability of the fth category in the true distribution; Q(f) represents the probability of the fth category in the model predicted distribution; Define the regression box loss function to measure the error between the network prediction box and the label box; The regression box loss function is: The value of E is p x -g x , p y -g y , p w -g w and p h -g h ; Among them, (p x , p y ) is the center coordinate of the prediction box, p w is the width of the prediction box, p h is the height of the prediction box; (g x , g y ) is the center coordinate of the label box, g w is the width of the label box, g h is the height of the label box; N is the number of images in the total image training set; The back propagation algorithm is used to update the network parameters according to the loss function, so that the prediction box is constantly close to the label box; the extraction network is finally obtained by iteratively optimizing the model; The steps of training and optimizing the network include: Use the extraction network to evaluate all candidate boxes generated by the training set images; evaluate the IoU between each candidate box and the label box; IoU indicates the degree of overlap between the two; If the IoU of the candidate box is greater than 0.5, the candidate box is marked as a positive sample; if the IoU of the candidate box is less than or equal to 0.5, the candidate box is marked as a negative sample; Define the network structure of the optimization network, which includes convolutional layers, pooling layers, and fully connected layers; define the training goal of the optimization network as outputting a binary judgment result, which is whether the current candidate box includes a face; expect the judgment result to be consistent with the annotated label; the label includes positive samples and negative samples; The binary cross entropy loss function is used as the loss function for optimizing the network; Binary cross entropy loss function Loss(o,y)=-(ylog(o)+(1-y)log(1-o)); Among them, o is the probability that the current sample is predicted to be a positive sample; y is the label of the current sample, which is 1 for positive samples and 0 for negative samples; Input the image with the candidate box to the optimized network for network calculation to obtain the binary judgment result; compare the binary judgment result with the label and calculate the value of the binary cross entropy loss function; update the network parameters of the optimized network through the back propagation algorithm. The value of the binary cross entropy loss function will gradually decrease during the update and optimization process. When its decrease rate slows down, the training is terminated; and the optimized network is obtained; The step of training the output network comprises: After filtering by the optimized network, the high-quality candidate frame images left are used as training samples for the output network; the network structure for constructing the output network is the same as that of the optimized network; The loss function of the output network is defined as a coordinate regression loss function, which is used to measure the sum of square errors between the output network prediction box and the label box in terms of position and size. The input of the output network is the candidate box image, and the output of the output network is the adjusted prediction box coordinates; the prediction box coordinates include the center coordinates of the prediction box, the width of the prediction box, and the height of the prediction box; Predefine the parameters of the output network, and the output network predicts the coordinates of the candidate box after the output adjustment according to the current parameters; calculate the coordinate regression loss function value of the predicted box and the image label box; calculate the gradient of each parameter to the coordinate regression loss function value by derivation by the chain rule; fine-tune the parameters of the output network along the negative gradient direction to reduce the value of the coordinate regression loss function; when the reduction speed slows down, end the training; and obtain the output network; After the three stages are trained, they are connected in series to form a complete MTCNN face detection model.
2. The intelligent design system for face replacement according to claim 1, It is characterized in that The extraction process of the facial key point coordinate sequence includes: Construct a key point detection convolutional neural network, which includes a convolutional layer, a pooling layer, and a fully connected layer; the input of the key point detection convolutional neural network is a face area image; the face area image is an RGB image; the output of the key point detection convolutional neural network is the coordinates of the face key points, and there are c face key points in total; Collect and annotate v facial image feature images to form a facial image feature image set; The collected facial image feature images cover different genders, ages, races and facial expressions and postures; the coordinates of c key points of the face in each facial image feature image are manually or computer marked using an image annotation tool; The labeled facial image feature set is input into the key point detection convolutional neural network; the loss function is the Euclidean distance between the predicted key point coordinates and the labeled key point coordinates; The loss function on a single face image feature image is in, The jth key point coordinates of the i-th image predicted by the key point detection convolutional neural network; y ij The coordinates of the jth key point manually annotated for the i-th image; To predict the key point coordinates And the coordinate y of the key point ij The Euclidean distance; c is the number of facial key points; On the face image feature image set, the overall loss function of the key point detection model is: Among them, v is the number of all images included in the face image feature image set; Define the network parameters of the key point detection convolutional neural network and initialize it; input the images in the face image feature set for training, and forward propagate to calculate the loss L z ; Calculate the loss L through the back propagation algorithm z The gradient of the parameter θ By gradient descent Among them, α is the learning rate; through iterative optimization, the loss L is gradually reduced. z The value of , get the face key point detection model; Input the face region image into the face key point detection model to predict and extract the output c key point coordinates; construct the face key point coordinate sequence from the c key point coordinates; The deep learning feature extraction process includes: Predefine the middle layer of the face key point detection model; input the face area image into the face key point detection model; extract the deep feature vector output by the middle layer of the face key point detection model; the deep feature vector is the deep learning feature; The method of acquiring the facial region features includes: The depth feature vector is a fixed-length vector, and the facial key point coordinate sequence is of variable length; the facial key point coordinate sequence is flattened, and each key point is represented by a vector of length 2; the depth feature vector is concatenated with the flattened facial key point coordinate sequence to obtain a fixed-length vector.
3. The intelligent design system for face replacement according to claim 2, It is characterized in that The method for obtaining the replacement matching degree includes: The facial feature extraction module is used to extract the facial region features of the face region image A to be replaced and the facial region features of the target face image B respectively; n pairs of facial region features are divided according to n sub-regions; the facial region features are high-dimensional vectors, including two parts of information: deep learning features and facial key point coordinate sequences; the cosine similarity sim(a s ,b s ); Among them, a s is the facial region feature of the face region image A to be replaced in the s-th sub-region; b s is the facial region feature of the target face image B in the sth sub-region; a s .b s Indicates a s and b s Inner product of vectors; ||a s || and ||b s || respectively represent a s and b s The L2 norm of ; sim(a s ,b s ) is used as the replacement matching degree between the face area image to be replaced and the target face image sub-area.
4. The intelligent design system for face replacement according to claim 3, It is characterized in that The matching threshold interval can be set in the following ways: Collect a training data set including u pairs of face images, each pair of face images is a face image of the same person taken in different scenes; calculate the matching score for each pair of face images ij ; Count the matching scores of all face images in the entire training data set ij , calculate the matching score of all face images ij The mean Calculate the matching score of all face images ij Standard Deviation Where u is the number of face image pairs in the training dataset; Set the upper limit of the matching threshold interval to ρ+3σ; set the lower limit of the matching threshold interval to ρ-3σ; then the matching threshold interval is [ρ+3σ,ρ-3σ]; The comparison result is a high match, a medium match, or a low match; If the replacement matching degree among the n replacement matching degrees is greater than or equal to, the comparison result of the corresponding sub-region is a high match; if the replacement matching degree among the n replacement matching degrees is within the matching degree threshold interval, the comparison result of the corresponding sub-region is a medium match; if the replacement matching degree among the n replacement matching degrees is less than or equal to ρ-3σ, the comparison result of the corresponding sub-region is a low match.
5. The intelligent design system for face replacement according to claim 4, It is characterized in that The method of using an image fusion algorithm according to the comparison result to fuse and replace the target portrait image and the face area image to be replaced includes: The target portrait image with a high match result and the sub-region of the face area to be replaced are fused and replaced using the Poisson fusion algorithm; The target portrait image and the sub-region of the face area to be replaced are fused and replaced by using a color migration algorithm; The target portrait image with a low match in comparison result and the sub-region of the face area image to be replaced are fused and replaced using a geometric transformation algorithm; and the fusion replacement between the target portrait image and the sub-region of the face area image to be replaced is achieved.
6. An intelligent design method for face replacement, which is implemented based on an intelligent design system for face replacement according to any one of claims 1 to 5. It is characterized in that include: S1, perform face detection on the input video stream to obtain a face area image; S2, extracting facial key point coordinate sequence and deep learning features from the facial region image, and obtaining facial region features according to the facial key point coordinate sequence and the deep learning features; S3, comparing facial region features of the facial region image to be replaced with the target facial image, dividing the facial region image to be replaced and the target facial image into n sub-regions, and obtaining n replacement matching degrees between the facial region image to be replaced and the n sub-regions of the target facial image; S4, setting a matching degree threshold interval, and obtaining a matching result by comparing n replacement matching degrees and the matching degree threshold interval; and using an image fusion algorithm according to the matching result to fuse and replace the target portrait image and the face area image to be replaced.
