Figure model face changing method and system based on deep learning
Through the face-changing method of character model based on deep learning, the user's facial features are combined with the upper body effect of the clothing, solving the problem that the existing technology cannot combine face image data for clothing display, and improving the effectiveness of the display and shopping experience.
Patent Information
- Application Number
- CN202411457619.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-14
AI Technical Summary
The existing clothing display system cannot combine facial image data for clothing display, resulting in the inability to match the clothing and the human face shape, which reduces the effectiveness of the display.
Using the face-changing method of character model based on deep learning, the face image of the user is obtained, and the facial feature map is recognized and extracted using a pre-trained convolutional neural network model, and these features are mapped to the three-dimensional model matching the target character to generate the user. A synthetic image of the fitting effect of the target character of facial features.
It realizes the better combination of user's facial features with the upper body effect of the clothing, improves the effectiveness of clothing display, enables users to purchase clothes more intuitively and improves the shopping experience.
Smart Images
Figure CN119445032A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology and equipment, and in particular to a method and system for changing the face of a character model based on deep learning. Background Art
[0002] Existing users have a variety of ways to display clothing when purchasing clothing, mainly including static picture display and 360-degree rotating display.
[0003] Static picture display is the most traditional and commonly used method, which uses professional photographers to take pictures of clothing from multiple angles to show the color, texture, design details and tailoring characteristics of the clothing. These pictures usually include the front, back, side and important details, so that consumers can fully understand the appearance and quality of the clothing.
[0004] The 360-degree rotating display is an upgrade based on static pictures. It is composed of a series of continuously taken photos into dynamic images or completed at one time using special photographic equipment. Customers can freely view every angle of the clothing by dragging the mouse or sliding their fingers on the touch screen. This display method not only provides a full range of visual effects, but also allows consumers to better estimate the actual wearing effect of the clothing, especially when showing the overall outline and streamline of the clothing.
[0005] However, the existing display system cannot combine facial image data for clothing display, so it is impossible to match clothing with people's faces, which reduces the effectiveness of the display. Summary of the invention
[0006] The purpose of the present invention is to provide a method and system for changing the face of a character model based on deep learning, aiming to better show the effect of clothes on the body by combining the user's facial data, so that the user can buy clothes more intuitively and improve the shopping experience.
[0007] To achieve the above-mentioned object, the present invention provides a method for face-changing a character model based on deep learning, comprising obtaining a facial image of a user;
[0008] Use a pre-trained convolutional neural network model to identify and extract facial feature maps from facial images;
[0009] Mapping the facial feature map onto a matching three-dimensional model of the target person;
[0010] Generate a synthetic image of the target person's clothing-trying effect that includes the user's facial features.
[0011] The specific steps of obtaining the user's facial image include:
[0012] Turn on the user's side camera when the user grants access rights;
[0013] Use facial recognition algorithms to detect the face position within the camera's field of view and trigger the photo function;
[0014] The captured image is cropped to retain only the face portion and aligned to a standard position and size.
[0015] The specific steps of using the face recognition algorithm to detect the face position in the camera field of view and trigger the photo function include:
[0016] Use Haar cascade classifier to scan each frame of image;
[0017] The photo-taking mechanism is triggered when one or more valid face positions are detected.
[0018] The specific steps of cropping the captured image to retain only the face portion and aligning it to a standard position and size include:
[0019] Identify feature points on a person's face, including eyes, nose, and mouth;
[0020] Calculate a transformation matrix based on the positions of the key points, and rotate the face to a standard angle alignment using the transformation matrix;
[0021] Scales the image to a preset size after alignment.
[0022] The specific steps of using the pre-trained convolutional neural network model to identify and extract facial feature maps in facial images include:
[0023] Train a convolutional neural network model on a facial dataset to recognize and classify different facial features;
[0024] Preprocessing the input facial image;
[0025] The preprocessed facial image is passed as input to the convolutional neural network model, which outputs a facial feature map.
[0026] The specific steps of training a convolutional neural network model on a facial dataset to identify and classify different facial features include:
[0027] Collect facial datasets;
[0028] Preprocess the face images in the face dataset;
[0029] Divide the preprocessed dataset into training set, validation set and test set;
[0030] Set the loss function, select the optimizer, and determine the learning rate, batch size, and training rounds to obtain the recognition model;
[0031] The recognition model is trained using the training set data and verified using the validation set to obtain a convolutional neural network model.
[0032] The specific steps of preprocessing the face images in the face data set include:
[0033] Resize the image to the desired fixed size;
[0034] Convert the image to grayscale;
[0035] Normalize the grayscale image.
[0036] The specific steps of mapping the facial features onto a three-dimensional model matching the target person include:
[0037] Obtain facial feature map data;
[0038] Match the corresponding 3D character model according to the user's body shape data;
[0039] The facial feature points extracted from the facial feature map are aligned with the corresponding points on the 3D model using an iterative closest point algorithm;
[0040] Map the texture of the face onto the 3D model.
[0041] In a second aspect, the present invention also provides a character model face-changing system based on deep learning, comprising an image acquisition module, a feature acquisition module, a mapping module and a synthesis module;
[0042] The image acquisition module is used to acquire the user's facial image;
[0043] The feature acquisition module is used to identify and extract facial feature maps in facial images using a pre-trained convolutional neural network model;
[0044] The mapping module is used to map the facial feature map onto a matching three-dimensional model of the target person;
[0045] The synthesis module is used to generate a synthetic image of the target person's fitting effect including the user's facial features.
[0046] The present invention provides a method and system for changing the face of a character model based on deep learning, and obtains the facial image of the user: the user provides one or more images containing his or her face through a camera, uploading photos or other means. These images need to clearly show the facial features of the user for subsequent processing. The facial image provided by the user is analyzed using a pre-trained convolutional neural network (CNN). This CNN model is usually trained with a large amount of facial data, and can accurately identify facial key points such as eyes, nose, and mouth, and extract facial contours, textures and other important facial features. The purpose of this stage is to obtain an accurate facial feature map, which will serve as the basis for the next step. After obtaining the user's facial feature map, the system will use graphics processing technology to map these features to a three-dimensional model corresponding to the target character (such as a movie character, a game character, etc.). This step may involve complex geometric transformations and texture mapping to ensure that the user's facial features can be naturally displayed on the three-dimensional model. Finally, by fusing the user's facial features onto the 3D model, the system can generate a new synthetic image or video sequence in which the user can see his or her face appear on the target person's body, thereby better showing how the clothes fit, allowing the user to purchase clothes more intuitively and improving the shopping experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 This is a flowchart of a method for replacing a character model face based on deep learning according to the first embodiment of the present invention.
[0049] Figure 2 FIG. 1 is a flowchart of obtaining a user's facial image according to the first embodiment of the present invention.
[0050] Figure 3 This is a flow chart of using a face recognition algorithm to detect the face position within the camera field of view and trigger a photo-taking function according to the first embodiment of the present invention.
[0051] Figure 4 It is a flowchart of the first embodiment of the present invention for cropping a captured image to retain only the face portion and aligning it to a standard position and size.
[0052] Figure 5It is a flowchart of using a pre-trained convolutional neural network model to recognize and extract facial feature maps in facial images according to the first embodiment of the present invention.
[0053] Figure 6 It is a flowchart of the first embodiment of the present invention for training a convolutional neural network model on a facial dataset to recognize and classify different facial features.
[0054] Figure 7 This is a flowchart of preprocessing face images in a face data set according to the first embodiment of the present invention.
[0055] Figure 8 It is a flowchart of mapping the facial features onto a matching three-dimensional model of a target person according to the first embodiment of the present invention. DETAILED DESCRIPTION
[0056] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0057] First embodiment
[0058] See also Figures 1 to 8 The present invention provides a method for changing the face of a character model based on deep learning, comprising:
[0059] S101 obtains a facial image of a user;
[0060] The specific steps include:
[0061] S201: when the user grants access permission, turn on the camera on the user side;
[0062] The app will ask the user for permission to access the camera on their device. The camera will only be activated if the user explicitly agrees and gives the appropriate permissions. This is to ensure that the user's privacy is protected and in compliance with relevant laws and regulations on data collection.
[0063] S202 uses a face recognition algorithm to detect the face position within the camera's field of view and trigger the photo function;
[0064] Once the camera is turned on, the system uses facial recognition technology to analyze each frame of the video stream. The algorithm looks for features of human faces and determines where faces are in the image. If one or more faces are detected, the system automatically triggers the camera function to capture the scene at that time. The specific steps include:
[0065] S301 uses Haar cascade classifier to scan each frame of the image;
[0066] Haar cascade classifier is a commonly used face detection method, which can effectively identify the facial area in the image. In this way, the system can quickly filter out the part containing the face.
[0067] S302 triggers a photo taking mechanism when one or more valid face positions are detected.
[0068] If the Haar cascade classifier successfully recognizes facial features, the system will capture the current frame as a facial image.
[0069] S203 crops the captured image to retain only the face portion, and aligns it to a standard position and size.
[0070] The specific steps include:
[0071] S401 identifies feature points on a person's face, where the feature points include eyes, nose, and mouth;
[0072] In order to locate and process the face more accurately, the system needs to identify key parts of the face such as the eyes, nose and mouth.
[0073] S402 calculates a transformation matrix based on the positions of the key points, and rotates the face to a standard angle alignment using the transformation matrix;
[0074] Using these landmarks, the system can calculate a transformation matrix that helps adjust the face to a standard angle, ensuring that the final image is properly aligned regardless of the orientation of the face originally captured.
[0075] S403 scales the image to a preset size after alignment.
[0076] Finally, the system will scale the aligned image to a preset size for subsequent processing or storage.
[0077] S102 uses a pre-trained convolutional neural network model to identify and extract facial feature maps from facial images;
[0078] The specific steps include:
[0079] S501 trains a convolutional neural network model on a facial dataset to recognize and classify different facial features;
[0080] At this stage, the goal is to create a model that can accurately identify facial features. To do this, you first need a dataset containing a large number of labeled facial images. The specific steps include:
[0081] S601 collects a facial data set;
[0082] These images should cover a variety of facial angles, expressions, lighting conditions, etc. to improve the generalization ability of the model.
[0083] S602 pre-processes the face images in the face data set;
[0084] The preprocessing step is to ensure that all input images are consistent and easy for the model to learn. This usually includes resizing the image, converting the color mode, and normalizing the value range. The specific steps include:
[0085] S701 adjusts the image size to a desired fixed size;
[0086] Since different images may have different resolutions, all images need to be unified into a fixed size for easy model processing.
[0087] S702 converts the image into a grayscale image;
[0088] Converting color images to grayscale can reduce the complexity of the model and in some cases help improve the performance of the model.
[0089] S703 normalizes the grayscale image.
[0090] Normalization is to standardize the image pixel values to a specific range, such as between 0 and 1, which can help the model converge faster and improve training efficiency.
[0091] S603 divides the preprocessed data set into a training set, a validation set, and a test set;
[0092] The purpose of data set division is to evaluate the performance of the model and prevent overfitting. The training set is used for model learning, the validation set is used to adjust hyperparameters, and the test set is used to finally evaluate the generalization ability of the model.
[0093] S604 sets a loss function, selects an optimizer, and determines a learning rate, a batch size, and a training round to obtain a recognition model;
[0094] Before model training, you need to define a loss function to measure the gap between the model prediction and the actual label, and choose a suitable optimizer to minimize this gap. At the same time, you also need to set some training parameters, such as learning rate, batch size, and training rounds.
[0095] S605 uses the training set data to train the recognition model and verifies it through the verification set, thereby obtaining a convolutional neural network model.
[0096] During the training process, the model will continuously adjust its weights according to the training set data to minimize the loss function. At the same time, the validation set can be used to monitor the performance of the model and avoid overfitting.
[0097] S502 pre-processes the input facial image;
[0098] During the model application phase, the input facial images also need to undergo the same preprocessing operations as in the training phase to ensure image consistency and model effectiveness.
[0099] S503 passes the preprocessed facial image as input to the convolutional neural network model, and the convolutional neural network model outputs a facial feature map.
[0100] Finally, the preprocessed facial image is fed into the trained convolutional neural network model for feature extraction. The model outputs a facial feature map, which contains important facial feature information and can be used for subsequent tasks such as face recognition or expression analysis.
[0101] S103 maps the facial feature map onto a matching three-dimensional model of the target person;
[0102] The specific steps include:
[0103] S801 obtains facial feature map data;
[0104] In this step, the system reads the facial feature map data previously output by the convolutional neural network model. This data usually contains a series of facial feature points and their relative position information, which is the basis for the next step of mapping to a three-dimensional model.
[0105] S802 matches a corresponding three-dimensional character model according to the user's body shape data;
[0106] Next, the system needs to select a suitable 3D character model based on the user's body shape data, which can include information such as height, weight, and shoulder width, to ensure that the 3D model not only matches the facial features but also has an overall appearance similar to the user.
[0107] S803 uses an iterative closest point algorithm to align facial feature points extracted from the facial feature map with corresponding points on the three-dimensional model;
[0108] The ICP algorithm is a commonly used point cloud registration algorithm that is used to find the best transformation between two point sets to minimize the distance between them. In this scenario, the system will use the ICP algorithm to align the feature points in the facial feature map with the corresponding feature points on the 3D model so that the positional relationship between the two is as consistent as possible.
[0109] S804 maps the texture of the face onto the three-dimensional model.
[0110] The final step is to map the facial texture information onto the aligned 3D model. This step will ensure that the 3D model not only has the correct geometry, but also has a realistic facial appearance. Texture mapping is a complex process that involves projecting a 2D image onto the surface of a 3D object to simulate real-world visuals. Through this step, the resulting 3D model will have realistic facial details.
[0111] S104 generates a composite image of the target person's fitting effect including the user's facial features.
[0112] First, based on the facial feature mapping completed in the previous step, the 3D model now already has the user's facial features, including but not limited to facial details such as eyes, nose, mouth, and skin texture and color.
[0113] Next, the system needs to load the 3D models of the clothing selected by the user. These clothing models are usually made based on actual clothing designs and can accurately reflect the material, color and style of the clothing. The system will place these clothing models in appropriate positions on the 3D character model to simulate the effect of trying on.
[0114] The system then performs lighting and shading on the 3D character model to ensure that the clothing and character model are visually consistent and realistic. This step may involve real-time rendering technology to ensure the quality and effect of the synthesized image, making the final image look more natural and realistic.
[0115] At the same time, the system also needs to consider the issue of viewing angle. In order to provide users with a full range of fitting experience, the system may generate composite images from multiple viewing angles, or allow users to freely rotate to view the fitting effects at different angles.
[0116] Finally, after all the preparations are completed, the system uses image synthesis technology to combine the 3D model with the background environment or other elements to generate the final composite image. This synthesis must not only take into account the visual consistency of the characters and clothing, but also ensure that the entire scene looks harmonious without obvious splicing marks.
[0117] The generated synthetic images will show a target person trying on clothes with the user's facial features. Users can use these images to intuitively see how they would look in the selected clothes, thus helping them make purchasing decisions.
[0118] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. A method for changing the face of a character model based on deep learning, characterized in that: include: Get the user's facial image; Use a pre-trained convolutional neural network model to identify and extract facial feature maps from facial images; Mapping the facial feature map onto a matching three-dimensional model of the target person; Generate a synthetic image of the target person's clothing-trying effect that includes the user's facial features.
2. A method for changing the face of a character model based on deep learning as claimed in claim 1, characterized in that: The specific steps of obtaining the user's facial image include: Turn on the user's side camera when the user grants access rights; Use facial recognition algorithms to detect the face position within the camera's field of view and trigger the photo function; The captured image is cropped to retain only the face portion and aligned to a standard position and size.
3. A method for changing the face of a character model based on deep learning as claimed in claim 2, characterized in that: The specific steps of using the face recognition algorithm to detect the face position in the camera field of view and trigger the photo function include: Use Haar cascade classifier to scan each frame of image; The photo-taking mechanism is triggered when one or more valid face positions are detected.
4. A method for changing the face of a character model based on deep learning as claimed in claim 3, characterized in that: The specific steps of cropping the captured image to retain only the face portion and aligning it to a standard position and size include: Identify feature points on a person's face, including eyes, nose, and mouth; Calculate a transformation matrix based on the positions of the key points, and rotate the face to a standard angle alignment using the transformation matrix; Scales the image to a preset size after alignment.
5. A method for changing the face of a character model based on deep learning as claimed in claim 4, characterized in that: The specific steps of using the pre-trained convolutional neural network model to identify and extract facial feature maps in facial images include: Train a convolutional neural network model on a facial dataset to recognize and classify different facial features; Preprocessing the input facial image; The preprocessed facial image is passed as input to the convolutional neural network model, which outputs a facial feature map.
6. A method for changing the face of a character model based on deep learning as claimed in claim 5, characterized in that: The specific steps of training a convolutional neural network model on a facial dataset to identify and classify different facial features include: Collect facial datasets; Preprocess the face images in the face dataset; Divide the preprocessed dataset into training set, validation set and test set; Set the loss function, select the optimizer, and determine the learning rate, batch size, and training rounds to obtain the recognition model; The recognition model is trained using the training set data and verified using the validation set to obtain a convolutional neural network model.
7. A method for changing the face of a character model based on deep learning as claimed in claim 6, characterized in that: The specific steps of preprocessing the face images in the face data set include: Resize the image to the desired fixed size; Convert the image to grayscale; Normalize the grayscale image.
8. A method for changing the face of a character model based on deep learning as claimed in claim 7, characterized in that: The specific steps of mapping the facial features to the three-dimensional model matching the target person include: Obtain facial feature map data; Match the corresponding 3D character model according to the user's body shape data; The facial feature points extracted from the facial feature map are aligned with the corresponding points on the 3D model using an iterative closest point algorithm; Map the texture of the face onto the 3D model.
9. A character model face-changing system based on deep learning, applied to a character model face-changing method based on deep learning as claimed in any one of claims 1 to 8, characterized in that: It includes an image acquisition module, a feature acquisition module, a mapping module and a synthesis module; The image acquisition module is used to acquire the user's facial image; The feature acquisition module is used to identify and extract facial feature maps in facial images using a pre-trained convolutional neural network model; The mapping module is used to map the facial feature map onto a matching three-dimensional model of the target person; The synthesis module is used to generate a synthetic image of the target person's fitting effect including the user's facial features.