Virtual dressing method, device, electronic device and storage medium

Abstract

The present application proposes a virtual dressing method, device, electronic device and storage medium, the method comprising: obtaining a full-body image of the user; identifying the coordinates of the user's facial key points and the coordinates of the user's body contour key points from the full-body image; creating a full-body three-dimensional model of the user based on the full-body image, the coordinates of the facial key points and the coordinates of the body contour key points; generating a fitting effect diagram based on the full-body three-dimensional model and the clothing parameter information selected by the user. The present application creates a real-life full-body three-dimensional model of the user, and recommends clothing based on the user's real facial features and body features. The fitting is completed on the full-body three-dimensional model through the three-dimensional picture and size data of the clothing, so that the clothing matches the full-body three-dimensional model in terms of height and face, avoids the clothing in the fitting effect diagram from being skewed and deformed, improves the referenceability of the fitting effect diagram, provides users with a more realistic virtual fitting experience, and also provides accurate data support and professional suggestions for user body management.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a virtual dressing method, device, electronic device and storage medium. Background Art

[0002] With the development of the economy, people pay more attention to dressing, and the demand for personalized and diversified dressing is getting stronger and stronger. At present, people often browse clothing pictures in online clothing stores to choose clothes, but there are often problems such as the actual clothing not matching the picture and the size not matching.

[0003] A virtual fitting method is proposed in the related art. The method obtains a user photo, selects the head reference point of the user photo, determines the user area based on the head reference point, and collects the reference point of the clothing image after the user clicks on the clothing image to be fitted. Based on the reference point of the clothing image and the determined user area, a finished dressing effect image is generated.

[0004] However, the related art is based on two-dimensional user photos and clothing pictures for changing clothes. The user's face shape, body shape and other features in the two-dimensional user photos are not realistic enough. When performing virtual fitting on the two-dimensional user photos, the clothes are easily skewed and deformed, resulting in poor reference value of virtual fitting. Summary of the invention

[0005] The present application proposes a virtual dressing method, device, electronic device and storage medium, which create a real-life full-body three-dimensional model of the user, complete clothing fitting on the full-body three-dimensional model, make the clothing and the full-body three-dimensional model match in terms of height and face, avoid clothing distortion in fitting renderings, improve the reference value of fitting renderings, and provide users with a more realistic virtual fitting experience.

[0006] The first aspect of the present application provides a virtual dressing method, the method comprising:

[0007] Obtaining a full-body image of the user;

[0008] Identify the coordinates of the user's facial key points and the coordinates of the user's body contour key points from the full-body image;

[0009] Creating a full-body three-dimensional model of the user according to the full-body image, the coordinates of the key points of the face, and the coordinates of the key points of the body contour;

[0010] A fitting effect diagram is generated according to the full-body three-dimensional model and the clothing parameter information selected by the user.

[0011] In some embodiments of the present application, creating a full-body three-dimensional model of the user according to the full-body image, the coordinates of the key points of the face, and the coordinates of the key points of the body contour includes:

[0012] Creating a three-dimensional facial model of the user according to the coordinates of the facial key points and the facial image included in the full-body image;

[0013] A three-dimensional body model of the user is created according to the coordinates of the key points of the human body contour, and a full-body three-dimensional model of the user is obtained.

[0014] In some embodiments of the present application, creating the three-dimensional face model of the user according to the coordinates of the facial key points and the facial image included in the full-body image includes:

[0015] According to the facial image included in the full-body image, identifying the gender category and hairstyle category of the user;

[0016] A three-dimensional facial model of the user is created according to the gender category, the hairstyle category and the coordinates of the facial key points corresponding to the user.

[0017] In some embodiments of the present application, creating the three-dimensional body model of the user according to the coordinates of the key points of the human body contour includes:

[0018] Predicting various human attributes of the user according to the coordinates of the key points of the human body contour;

[0019] A three-dimensional body model of the user is created based on the various body attributes and the coordinates of the key points of the body contour.

[0020] In some embodiments of the present application, before generating the fitting effect diagram according to the full-body three-dimensional model and the clothing parameter information selected by the user, the method further includes:

[0021] Acquiring clothing parameter information matching the user according to the full-body image;

[0022] The matching clothing parameter information is recommended to the user.

[0023] In some embodiments of the present application, after generating a fitting effect diagram according to the full-body three-dimensional model and the clothing parameter information selected by the user, the method further includes:

[0024] Generating dressing effect information corresponding to the clothing parameter information according to the full-body three-dimensional model and the clothing parameter information;

[0025] Display the costume effect information.

[0026] In some embodiments of the present application, the method further includes:

[0027] Receive body data input by the user;

[0028] The whole body three-dimensional model is modified according to the body shape data.

[0029] In some embodiments of the present application, the method further includes:

[0030] generating an appearance report of the user according to the facial image included in the full-body image;

[0031] generating a body report of the user according to the body image included in the full-body image;

[0032] A skin report of the user is generated according to the facial image and the body image.

[0033] An embodiment of a second aspect of the present application provides a virtual dressing device, the device comprising:

[0034] An acquisition module, used to acquire a full-body image of a user;

[0035] A recognition module, used to recognize the coordinates of the key points of the face and the key points of the body contour of the user from the full-body image;

[0036] A creation module, used for creating a full-body three-dimensional model of the user according to the full-body image, the coordinates of the key points of the face and the coordinates of the key points of the body contour;

[0037] A generation module is used to generate a fitting effect diagram according to the full-body three-dimensional model and the clothing parameter information selected by the user.

[0038] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0039] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0040] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0041] The embodiment of the present application identifies the coordinates of the key points of the user's face and body based on the user's full-body image, and creates a real-life full-body three-dimensional model of the user based on this, and can recommend clothing suitable for the user based on the user's real facial features and body features. When the user selects clothing from the recommended clothing for fitting, the clothing or accessories are fitted on the user's full-body three-dimensional model through the three-dimensional image and size data of the clothing, so that the clothing to be fitted matches the user's full-body three-dimensional model in terms of height and face, avoiding the problem of clothing skewness and deformation in the generated fitting effect picture, improving the referenceability of the fitting effect picture, and providing users with a virtual fitting experience that is more in line with reality. Furthermore, the embodiment of the present application also generates a face report, a body report, and a skin report for the user based on the user's real facial image and body image, providing accurate data support and professional body management suggestions for the user's body management.

[0042] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0044] In the attached picture:

[0045] Figure 1 A flowchart of a virtual dressing method provided by an embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram of an interface for creating a three-dimensional facial model provided in an embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of an interface for creating a full-body three-dimensional model provided in an embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram of the structure of a dual-attribute recognition model provided by an embodiment of the present application is shown;

[0049] Figure 5 A schematic diagram of the structure of a residual module provided in an embodiment of the present application is shown;

[0050] Figure 6 A schematic diagram of the structure of a depth-separable convolution module provided in another embodiment of the present application is shown;

[0051] Figure 7A schematic diagram showing key points of a human body contour provided by another embodiment of the present application is shown;

[0052] Figure 8 A schematic diagram showing an indicator for evaluating any human attribute in the second category of human attributes provided by another embodiment of the present application is shown;

[0053] Fig. 9 Another schematic diagram showing an indicator for evaluating any human attribute in the second category of human attributes provided by another embodiment of the present application;

[0054] Fig.10 A schematic diagram showing the structure of a preset neural network prediction model provided in another embodiment of the present application is shown;

[0055] Fig.11 A schematic diagram of an interface for manually inputting body shape data to modify a full body three-dimensional model provided in another embodiment of the present application is shown;

[0056] Fig.12 A schematic diagram of an interface of an appearance report, a body report and a skin report provided by another embodiment of the present application is shown;

[0057] Fig.13 A schematic diagram of an interface for performing virtual dressing on a full-body three-dimensional model provided by another embodiment of the present application is shown;

[0058] Fig.14 A schematic diagram of the structure of a virtual dressing device provided in an embodiment of the present application is shown;

[0059] Fig.15 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0060] Fig.16 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0061] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0062] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0063] The following describes a virtual dressing method, device, electronic device and storage medium proposed in an embodiment of the present application in conjunction with the accompanying drawings.

[0064] At present, people have an increasing demand for personalized and diversified dressing. Some virtual fitting solutions have been proposed in related technologies. These solutions obtain user photos, select the head reference points of the user photos, determine the user area based on the head reference points, and collect the reference points of the clothing pictures after the user clicks on the clothing pictures to be tried on. Based on the reference points of the clothing pictures and the determined user areas, the effect pictures after dressing are generated, so that users can choose their favorite clothes online.

[0065] However, the above-mentioned related technologies are based on two-dimensional user photos and clothing pictures for changing clothes. The user's face shape, body shape and other features in the two-dimensional user photos are not realistic enough, and the clothes in the generated renderings are easily skewed and deformed, resulting in low reference value for virtual fitting.

[0066] Based on this, the embodiment of the present application proposes a virtual dressing method, which identifies the key points of the user's face and the key points of the body contour according to the user's full-body image, and creates a three-dimensional face model and a three-dimensional body model respectively, thereby obtaining a three-dimensional full-body model of the user. The full-body three-dimensional model obtained in this way is closer to the user's real image. When performing virtual dressing on the full-body three-dimensional model, the clothes or accessories in the generated fitting effect picture are not easy to be skewed or deformed, and the user's own facial shape and body characteristics are fully considered, which greatly improves the reference value of the fitting effect picture, makes the virtual dressing more in line with the user's own reality, and provides the user with a more realistic dressing experience.

[0067] See also Figure 1 , the method specifically comprises the following steps;

[0068] Step 101: Acquire a full-body image of the user.

[0069] The execution subject of the embodiment of the present application is a terminal, such as a mobile phone, a tablet computer, a desktop computer, etc. The user's face image can be obtained by scanning the user's face through the camera configured on the terminal, and the user's body image can be obtained by scanning the user's body through the camera, thereby obtaining the user's full-body image. The full-body image including the user's face image and body image can also be directly obtained from the local gallery of the terminal.

[0070] In an embodiment of the present application, an application for virtual dressing is installed on the terminal. When a user needs to perform a virtual dressing, the user can log in to the application on the terminal, and the terminal displays a modeling interface after logging in. When the terminal detects that the user clicks on the modeling interface, the camera on the terminal is called to scan the user's face, and the effective facial scanning frame and facial posture prompt information are displayed during the scanning process. The facial posture prompt information is used to prompt the user to take off glasses, masks, lift up bangs, etc. to keep the face unobstructed, prompt the user to keep shooting vertically and not tilt; and to prompt the user to keep his face within the effective facial scanning frame during the scanning process. The terminal scans the user's face through the camera to obtain the user's facial image.

[0071] like Figure 2 As shown in the figure, the terminal takes a mobile phone as an example. After logging into the application for virtual dress-up on the mobile phone, an interface (1) is displayed. The interface (1) includes a modeling interface "Start generating my 3D model". After the user clicks "Start generating my 3D model", the mobile phone turns on the camera to scan the user's face and displays Figure 2 The interface (2) in the figure is displayed. The effective facial scanning frame is similar in shape to the outline of a human head. The interface (2) also displays a facial posture prompt message "Please lift your bangs, take off your glasses, and keep your face unobstructed." The user follows the prompt to align his or her face with the effective facial scanning frame for scanning. If the phone is detected to be tilted during the scanning process, the facial posture prompt message "The phone is tilted, please shoot vertically" is displayed. Figure 2 The interface (3) is shown in the figure.

[0072] The facial image obtained by camera scanning conforms to the actual facial condition of the user at present, and the facial image can reflect the actual condition of the user's current appearance, hairstyle, skin, etc.

[0073] In the embodiment of the present application, the user's body can also be scanned by the camera on the terminal to obtain the user's body image. Before scanning the body, the terminal displays the options of height and weight, and the user submits his height and weight respectively through the options of height and weight. Then the camera on the terminal scans the user's body, and the body effective scanning frame and body posture prompt information are displayed during the scanning process. The body posture prompt information is used to prompt the user to stand upright, stretch the arms, keep the body within the effective scanning frame of the body, etc. The terminal scans the user's body through the camera to obtain the user's body image.

[0074] like Figure 3As shown in , the terminal takes a mobile phone as an example. Since it is difficult for users to take a full-body photo with their mobile phones, the terminal displays the prompt information in interface (a) "Ask your friends to help take a beautiful full-body photo and start testing your body shape", and displays "Click anywhere on the screen to start shooting". When the terminal detects that the current screen is clicked, it enters interface (b), and displays a valid body scan frame in interface (b). The shape of the valid body scan frame is similar to the outline of a human body. Interface (b) also displays a body posture prompt information "Stand upright with your arms straight". The user follows the prompt to align his or her body with the valid body scan frame for scanning, thereby obtaining the user's body image.

[0075] The body image obtained by camera scanning conforms to the actual body condition of the user, and the body image can reflect the actual state of the user's current body condition.

[0076] Step 102: Identify the coordinates of the user's facial key points and the coordinates of the body contour key points from the full-body image.

[0077] Specifically, the coordinates of the key points can be identified by the following operations of steps A1 and A2, including:

[0078] A1: Based on the facial image included in the full-body image, identify the coordinates of the facial key points corresponding to the user.

[0079] The embodiment of the present application does not limit the number of key points of the face, as long as all key points of the face can fully cover the user's facial features, such as key points of the face including points on the user's facial contour and points on the ears, nose, mouth, eyes, eyebrows, etc. Figure 2 The interface (4) in FIG. 4 shows some facial key points of the user's facial image.

[0080] In the embodiment of the present application, the coordinates of the key points of the face in the face image are identified by a preset detection algorithm. The preset detection algorithm can use a detection framework such as CenterNet. On the basis of the original CenterNet, the basic network structure is replaced with EfficientNet-B0, which can achieve a good balance between accuracy and speed. The overall network structure of the preset detection algorithm is composed of EfficientNet-B0 plus a deconvolution module. The EfficientNet-B0 part consists of a normal convolution layer plus 7 MBconv (mobile inverted bottleneck convolution).

[0081] In the EfficientNet-B0 part, the size of the input facial image is 512*512*3 (height, width, channel). After 5 downsampling (width and height reduction) operations, the final output feature map size of the module is 16*16*320 (512 / 25=16). Then it goes through 3 deconvolution modules, each deconvolution will upsample the input feature map (width and height increase) once, and the final output feature map size after 3 deconvolutions is 128*128*256. On this basis, after a 1*1 convolution process, the final 6 network output results are obtained.

[0082] The output results of the above 6 networks are as follows:

[0083] Hm: feature map of the object, one channel (thickness) for one category. Since the detection method of the embodiment of the present disclosure only detects one category, the size of this layer is 128*128*1.

[0084] Wh: The size of the object, that is, width and height, so it is 2 channels and the final size is 128*128*2.

[0085] Hps: The key points of the object are based on the offset of the center point, and k*2 channels are output. k represents the number of key points. Assuming that in a certain application scenario, the human body contour key points of the target object are 77 human body contour key points, 2 represents the x and y offsets, and the final size is 77*2=154.

[0086] Reg: The offset of the center point of the body, including the x and y offsets. Therefore, it is 2 channels and the final size is 128*128*2.

[0087] hm_hp: feature map of the key points of the object, outputs k channels, one channel for each point, and the final size is 128*128*77.

[0088] hp_offset: The offset of K key points. All these key points use the same offset, two indicators x and y, output 2 channels, and the final size is 128*128*2.

[0089] After obtaining the above 6 network output results, use formula (1) to convert them into the final coordinates.

[0090] (x i +δx i -w i / 2,y i +δy i -h i / 2,x i +δxi +w i / 2,y i +δy i +h i / 2)…(1)

[0091] In formula (1), (x i ,y i ) represents the predicted center point x,y coordinates, i.e., hm branch. (δx i ,δy i ) represents the offset of the predicted center point coordinates x, y, i.e., the reg branch. (w i ,h i ) represents the size width and height of the object box.

[0092] The accurate coordinates of the center point are obtained by adding the offset of the center point coordinates in the x and y directions. Then, half of the width and height are subtracted from the center point coordinates to obtain the upper left corner coordinates of the target frame of the facial key points (x1, y1). The lower right corner coordinates of the target frame of the facial key points are obtained by adding half of the width and height to the center point coordinates (x2, y2).

[0093] Use formula (2) to convert to the coordinates of the final facial key points:

[0094]

[0095] In formula (2), l j is the coordinate of the jth facial key point, (x, y) represents the coordinate of the center point of the target box of the facial key point, that is, the hm branch. Represents the offset of the (x,y) coordinates of the jth point, that is, the hps branch.

[0096] In order to reduce the error of the coordinates of the recognized facial key points, the approximate position of each facial key point is determined by direct regression, and then the point Lj (hm_hp) with the highest confidence among the points near this position with a confidence (0-1 score) greater than 0.1 is found and taken as the real facial key point; then the point is used to add the offset (hp_offset) to obtain the final coordinates of the facial key points. That is, the coordinates of the final facial key points are calculated by formula (3).

[0097] L j =hm_hp+hp_offset…(3)

[0098] It should be noted that if there are more than two points with confidence greater than 0.1, the distance l is taken j The nearest point is as shown in formula (4):

[0099]

[0100] In formula (4), l j represents the coordinates of the key points of the face obtained based on the direct regression method, that is, the hps branch. Lj represents the coordinates of the key points of the face obtained based on the feature map method. Take the l corresponding to each key point of the face j Calculate the distance with Lj, and take the point in Lj with the closest distance as the final predicted face key point. For example, predict the key point position of the eye position based on regression, and then find the point with the closest confidence level higher than 0.1 in the feature map of the channel where the eye is located as the final eye position key point.

[0101] A2: According to the body image included in the full-body image, the coordinates of the key points of the human body contour corresponding to the user are identified.

[0102] The embodiments of the present application do not limit the number of key points of the human body contour, as long as the key points of the human body contour can fully cover the key parts of the user's body contour. The key parts of the body contour may include fingers, wrists, elbows, shoulders, neck, chest, stomach, waist, thighs, knees, calves, ankles, feet, etc. Figure 3 The interface (c) in Figure 1 marks some key points of the user's body contour. Figure 7 A schematic diagram of the key points of the human body contour is also given.

[0103] The method for identifying the coordinates of the key points of the human body contour is the same as the method for identifying the coordinates of the key points of the face in step A1, and the coordinates of the key points of the human body contour corresponding to the user are also identified by a preset detection algorithm. The size of the user's body image is adjusted to 512*512, and then input into the preset detection algorithm to obtain 6 network output results. According to the network output results, the center point coordinates of the target frame of each key point of the human body contour are calculated respectively by the above formula (1). Then, according to the center point coordinates of the target frame of each key point of the human body contour and the offset of the center point, the coordinates of each key point of the human body contour are calculated by formula (2). Furthermore, the coordinates of each key point of the human body contour can also be calculated by formulas (3) and (4) to reduce the error of the coordinates of the key points of the human body contour.

[0104] Step 103: Create a full-body three-dimensional model of the user based on the full-body image, the coordinates of the key points of the face, and the coordinates of the key points of the body contour.

[0105] Specifically, the user's full body three-dimensional model is created through the following operations A3 and A4, which specifically include:

[0106] A3: Create a 3D facial model of the user based on the coordinates of the facial key points and the facial image included in the full body image.

[0107] First, based on the facial image included in the full-body image, the gender category and hairstyle category of the user are identified.

[0108] The embodiment of the present application uses a pre-trained dual-attribute recognition model to perform gender recognition and hairstyle recognition on a user's facial image. The training process of the dual-attribute recognition model includes: obtaining a training data set, which includes head images labeled with gender categories and hairstyle categories; training a preset deep learning network with the training data set to obtain a trained dual-attribute recognition model.

[0109] The preset deep learning network may be mobilenet v2, VGG (Visual Geometry Group Network), ResNet (Residual Network), etc. The penultimate layer of the preset deep learning network includes two parallel fully connected layers, one of which is used for hairstyle recognition, and the other is used for gender recognition.

[0110] The structure of the trained dual-attribute recognition model is as follows: Figure 4 As shown, the dual attribute recognition model includes a fifth convolution layer, a first residual module, a first depth-separable convolution module, a second depth-separable convolution module, a third depth-separable convolution module, a second residual module, a fourth depth-separable convolution module, a third residual module, a sixth convolution layer, a pooling layer, a first fully connected layer, and a second fully connected layer, which are connected in sequence. The first fully connected layer and the second fully connected layer are two parallel branches, the first fully connected layer is used for hairstyle recognition, and the second fully connected layer is used for gender recognition.

[0111] Among them, the structures of the first residual module, the second residual module and the third residual module are as follows Figure 5 As shown, it includes the first convolution layer, the first depth-wise separable convolution layer, the second convolution layer and the addition operation layer connected in sequence. Among them, the convolution kernel size of the first convolution layer is 1×1, the activation function used is Relu6 (Rectified Linear Units), the step size is 1, and the zero padding is 0. The convolution kernel size of the first depth-wise separable convolution layer is 3×3, the activation function used is Relu6, the step size is 1, and the zero padding is 1. The convolution kernel size of the second convolution layer is 1×1, the activation function used is Linear, the step size is 1, and the zero padding is 0. The addition operation layer is used to add the output result of the second convolution layer to the initial input data of the depth-wise separable convolution module.

[0112] The structures of the first depth-separable convolution module, the second depth-separable convolution module, the third depth-separable convolution module and the fourth depth-separable convolution module are as follows: Figure 6As shown, it includes the third convolution layer, the second depth-wise separable convolution layer and the fourth convolution layer. Among them, the convolution kernel size of the third convolution layer is 1×1, the activation function used is Relu6, the step size is 1, and the zero padding is 0. The convolution kernel size of the second depth-wise separable convolution layer is 3×3, the activation function used is Relu6, the step size is 2, and the zero padding is 1. The convolution kernel size of the fourth convolution layer is 1×1, the activation function used is Linear, the step size is 1, and the zero padding is 0. From Figure 6 It can be seen intuitively that the residual module includes a second depth-wise separable convolution layer. Applying the depth-wise separable convolution to the residual structure can compress the size of the entire network model and reduce the amount of computation.

[0113] After the trained dual-attribute recognition model is obtained in the above manner, the dual-attribute recognition model can be used to perform gender recognition and hairstyle recognition on the user's facial image.

[0114] First, the facial image is preprocessed. Specifically, the size of the facial image is adjusted to a preset size through the image processing library opencv. The preset size can be 512×512 pixels or 1024×1024 pixels. Then the pixel value of each pixel in the facial image is divided by a preset value to obtain a normalized image corresponding to the facial image. The preset value can be 232 or 255. Then, each color channel of the normalized image is standardized according to the preset color mean and the preset color standard deviation to obtain a standardized image corresponding to the facial image. Normalizing and standardizing facial images can reduce the amount of image data, speed up calculations, and improve chest circumference recognition efficiency.

[0115] The preset color mean can be [0.485, 0.456, 0.406], where 0.485, 0.456, and 0.406 represent the mean values ​​of the red, green, and blue color channels in the ImageNet image library, respectively. For example, 0.485 is the average value of the pixel values ​​of all red pixels in the ImageNet image library, that is, the ratio of the sum of the pixel values ​​of all red pixels in the ImageNet image library to the total number of red pixels. The preset color standard deviation can be [0.229, 0.224, 0.225], where 0.229, 0.224, and 0.225 represent the standard deviations of the red, green, and blue color channels in the ImageNet image library, respectively. For example, 0.229 is the standard deviation of the pixel values ​​of all red pixels in the ImageNet image library.

[0116] The formula for normalizing each color channel of the normalized image is output = (input-mean) / std. Among them, output is the normalized pixel value, input represents the pixel value of a channel after normalization, mean represents the mean of the corresponding channel in the imagenet image library, and std represents the standard deviation of the corresponding channel in the imagenet image library. For example, assuming that the pixel value of the red channel in the normalized image is 0.5, then the standardized pixel value is (0.5-0.485) / 0.229=0.065.

[0117] After normalizing and standardizing the facial image in the above manner, the standardized image corresponding to the facial image is input into the trained dual-attribute recognition model, and the dual-attribute recognition model is used to perform gender recognition and hairstyle recognition on the facial image to obtain the classification result corresponding to the facial image. The classification result includes the gender score of the facial image belonging to each gender category and the hairstyle score of the facial image belonging to each hairstyle category.

[0118] The embodiment of the present application distinguishes hairstyle categories according to gender, such as dividing female hairstyles into long curly, long straight, medium straight, medium curly, short straight, short curly, tied up, etc., and dividing male hairstyles into short, medium, long, etc. After the dual-attribute recognition model outputs the gender scores and hairstyle scores of the facial image, the gender category with the highest gender score is determined as the gender category to which the facial image belongs. According to the hairstyle scores corresponding to each hairstyle category, the hairstyle category with the highest hairstyle score is obtained from the hairstyle categories corresponding to the determined gender categories. The hairstyle category with the highest hairstyle score is determined as the hairstyle category to which the facial image belongs.

[0119] For example, w represents female, m represents male, and c1, c2, c3, c4, c5, c6, c7, c8, c9, c10 represent long curly, long straight, medium straight, medium curly, short straight, short curly, tied up, short, medium, long. The gender score output by the dual-attribute recognition model is w=0.9, m=0.1, and the scores of each hairstyle are c1=0.91, c2=0.1, c3=0.1, c4=0.1, c5=0.1, c6=0.1, c7=0.1, c8=0.1, c9=0.1, c10=0.1. Then, according to the gender score, the gender category to which the facial image belongs is determined to be female w, and the hairstyle category "long curly" corresponding to the highest hairstyle score c1=0.91 is selected from the female hairstyles c1, c2, c3, c4, c5, c6, and c7 as the hairstyle category to which the facial image finally belongs.

[0120] Hairstyle categories are distinguished by gender, and a deep learning network is trained through a large number of images with gender and hairstyle annotated, so as to train a dual-attribute recognition model that can simultaneously identify gender and hairstyle. Combining gender recognition with hairstyle recognition and using gender recognition to assist hairstyle recognition can avoid the problem of misidentification of male and female hairstyles and improve the accuracy of hairstyle recognition.

[0121] In the embodiment of the present application, before performing gender recognition and hairstyle recognition on the user's facial image through the dual-attribute recognition model, the hairstyle of the user's facial image can also be recognized and the hair area in the facial image can be detected through a pre-trained single-attribute recognition model.

[0122] The training process of the single attribute recognition model includes: obtaining a training data set, which includes head pictures marked with hairstyle categories and coordinates of hair areas. The division of hairstyle categories when training the single attribute recognition model is the same as the division of hairstyle categories when training the dual attribute recognition model. For example, the hairstyles are divided into ten hairstyles: long curly, long straight, medium straight, medium curly, short straight, short curly, tied up, short, medium, and long. The coordinates of the hair area are (x, y, h, w). Among them, x, y are the center coordinates of the hair area (the units of x, y are pixels), and h, w are the height and width of the hair area (the units of h, w are also pixels).

[0123] The deep learning network is trained by the above training data set. The deep learning network selected here can be Efficientdet, VGG, ResNet, etc. The training process of the single attribute recognition model is similar to that of the dual attribute recognition model, which will not be repeated here.

[0124] After obtaining the single attribute recognition model in the above manner, the standardized image corresponding to the user's facial image is input into the trained single attribute recognition model, and the single attribute recognition model detects the hair area in the facial image and determines the coordinates (x, y, h, w) of the hair area. The single attribute recognition model performs hairstyle recognition on the detected hair area image, outputs the hairstyle score of each hairstyle category, and selects the hairstyle category with the highest hairstyle score as the hairstyle category finally recognized by the single attribute recognition model.

[0125] According to the coordinates of the hair area output by the single attribute recognition model, a hair area picture is cut out from the facial image. The hair area picture is used as a new facial image, and the gender recognition and hairstyle recognition of the new facial image are performed through the pre-trained dual attribute recognition model to obtain the gender score of the facial image and the hairstyle scores corresponding to each hairstyle category. The gender category with the highest gender score is determined as the gender category to which the facial image belongs. According to the hairstyle scores corresponding to each hairstyle category, the hairstyle category with the highest hairstyle score is obtained from the hairstyle categories corresponding to the determined gender categories. The hairstyle category with the highest hairstyle score is determined as the hairstyle category finally recognized by the dual attribute recognition model.

[0126] From the hairstyle categories identified by the single-attribute recognition model and the hairstyle categories identified by the dual-attribute recognition model, the hairstyle category with the highest hairstyle score is selected as the hairstyle category to which the facial image finally belongs.

[0127] The hairstyle recognition is performed on the image to be recognized through the single attribute recognition model and the dual attribute recognition model respectively, and the hairstyle category with the highest hairstyle score is selected from the hairstyle categories recognized by the two models as the hairstyle category to which the image to be recognized finally belongs, which further improves the accuracy of hairstyle recognition. In addition, the single attribute recognition model can detect the coordinates of the hair area in the image to be recognized, intercept the hair area image according to the coordinates of the hair area, and then use the dual attribute recognition model to perform gender recognition and hairstyle recognition on the hair area image, which can reduce the amount of data processed by the dual attribute recognition model and improve the efficiency of hairstyle recognition.

[0128] After the gender category and hairstyle category of the user are identified in the above manner, a three-dimensional facial model of the user is created according to the gender category, hairstyle category and coordinates of key points of the face corresponding to the user. Specifically, an affine transformation in three-dimensional space is performed according to the coordinates of key points of the face to generate a three-dimensional facial model. Then, according to the gender category and hairstyle category of the user, hairstyle parameter information corresponding to the gender category and hairstyle category of the user is obtained from a preset hairstyle library, and the three-dimensional facial model of the user is created according to the hairstyle parameter information and the generated three-dimensional facial model.

[0129] The preset hairstyle library includes hairstyle parameter information for each hairstyle category for men and hairstyle parameter information for each hairstyle category for women. The hairstyle categories for men include short, medium, long, etc., and each hairstyle category corresponds to hairstyle parameter information for multiple hairstyles. The hairstyle categories for women include long curly, long straight, medium straight, medium curly, short straight, short curly, tied up, etc., and each hairstyle category also corresponds to hairstyle parameter information for multiple hairstyles. The hairstyle parameter information includes the center coordinates of the hairstyle, the coordinates of the key points of the hairstyle's outline, the color of the hairstyle, the hairstyle thumbnail, etc.

[0130] The embodiment of the present application can randomly obtain a piece of hairstyle parameter information from the hairstyle parameter information corresponding to the gender category and hairstyle category in the preset hairstyle library according to the gender category and hairstyle category of the user. The hairstyle thumbnails of all the hairstyle parameter information corresponding to the gender category and hairstyle category can also be obtained from the preset hairstyle library. All the obtained hairstyle thumbnails are displayed so that the user can independently select a hairstyle that suits his actual situation or select a hairstyle he likes. When it is detected that the user clicks on a displayed hairstyle thumbnail, the hairstyle parameter information corresponding to the hairstyle thumbnail is obtained from the preset hairstyle library.

[0131] After obtaining the hairstyle parameter information in the above manner, a three-dimensional model of the hairstyle is rendered on the three-dimensional model of the face generated above according to the hairstyle parameter information including the center coordinates of the hairstyle, the coordinates of the key points of the hairstyle's contour, the color of the hairstyle, etc., to obtain a three-dimensional model of the user's face.

[0132] A4: Create a three-dimensional body model of the user according to the coordinates of the key points of the human body contour, and obtain a three-dimensional full-body model of the user.

[0133] In an embodiment of the present application, the user's body image and facial image can be obtained simultaneously in the manner of step 101, or the user's body image can be obtained in the manner of step 101 after creating a three-dimensional facial model of the user through the operation of the above-mentioned step A3.

[0134] like Figure 2 The interface (5) shown in the figure displays "Step 1: 3D face created successfully", and the interface (5) displays "Step 2: Create 3D body information", and displays options for height and weight. The user can submit his or her height through the height option and submit his or her weight through the weight option. Then click on the "Generate 3D Model" interface displayed in the interface (5) to scan your body to obtain a body image. Alternatively, you can also click on the "Upload body photo and generate exclusive 3D model" interface shown in the figure to directly select a body image from the local gallery to upload. If it is detected that the user has clicked on the "Generate 3D Model" interface, the display will jump to the next step. Figure 3 Interface (a) shown. When the terminal detects that the user clicks on interface (a), it enters interface (b), in which the effective body scan frame and body posture prompt information "Stand upright with arms straight" are displayed. The user follows the prompt to align his body with the effective body scan frame for scanning, thereby obtaining the user's body image.

[0135] After obtaining the user's body image in the above manner, identifying the coordinates of the user's body contour key points in step 102, and creating the user's facial three-dimensional model in step A3, the user's various body attributes are predicted based on the coordinates of the body contour key points.

[0136] In the embodiment of the present application, human attributes are divided into two categories. The first category of human attributes includes the user's height, chest circumference, underbust circumference, waist circumference, belly circumference, hip circumference, hip width, hip circumference, shoulder length, arm length, arm circumference, thigh circumference, calf circumference, leg length, head length, length from shoulder to base of thumb, vertical length from shoulder to crotch line, and shoulder width, etc.

[0137] Among them, the first category of human attributes can be divided into indicators based on length and indicators based on circumference. Indicators based on length include: height, hip width, shoulder length, arm length, leg length, head length, length from shoulder to base of thumb, vertical length from shoulder to crotch, and shoulder width. Indicators based on circumference include: chest circumference, underbust circumference (for women), waist circumference, belly circumference, hip circumference, hip circumference, arm circumference, thigh circumference, and calf circumference.

[0138] For the estimation based on the length metric, the pupil distance is used as the standard scale in the image. Generally, the pupil distance of men is between 60 mm and 73 mm, and the pupil distance of women is generally between 53 mm and 68 mm. The pupil distance can be taken as 66.5 for men and 60.5 for women. For women, the size of the 9 first-category human attributes based on length can be calculated by formula (5). For men, the size of the 9 first-category human attributes based on length can be calculated by formula (6).

[0139]

[0140]

[0141] For example, if the gender category of the user is identified as female in step 102, the product of the user's height in the user's body image and 60.5 is calculated, and then the ratio between the product and the user's pupil distance in the user's face image is calculated, and the ratio is the user's real height.

[0142] In the embodiment of the present application, the calculation of the length-based index and the index relationship of the corresponding human body contour key points are shown in Table 1, where L represents the length and p represents Figure 7 The key points of the human body outline marked in the figure, midpoint means the midpoint. For example, p6 means Figure 7 The key point of the human body contour marked with number 6, midpoint (p37, p47) means Figure 7 The midpoint of the line between the human body contour key points 37 and 47 marked in the figure, L(p6,midpoint(p37,p47)) represents the length between the human body contour key point 6 and the midpoint of the line between the human body contour key points 37 and 47, which is the height.

[0143] Table 1

[0144]

[0145] For the estimation of the circumference-based indicators, the elliptical model is used to fit the chest circumference, underbust circumference (for women), waist circumference, belly circumference, hip circumference, and hip circumference. The circumference formula of the elliptical model is formula (7), where a and b represent the major and minor semi-axis lengths of the ellipse, respectively.

[0146] L=2πb+4(ab)…(7)

[0147] Since the embodiment of the present application uses a frontal body image of the user, only the frontal indicator length can be measured, for example, the chest circumference can only measure the major semi-axis length. For the minor semi-axis length, statistics are performed based on the data of a preset number of real users to obtain the numerical relationship between the minor semi-axis length and the major semi-axis length. The preset number can be 1000 or 2000, etc. Among them, for each circumference-based indicator, the numerical relationship between the major semi-axis and the minor semi-axis is shown in Table 2.

[0148] Table 2

[0149]

[0150]

[0151] In the embodiment of the present application, a circular model is used to fit the arm circumference, thigh circumference, and calf circumference. The circumference formula of the circular model is formula (8), where d represents the diameter length of the circle.

[0152] L=πd…(8)

[0153] Table 3 shows the index relationship between the hip circumference, thigh circumference and calf circumference and the corresponding human body contour key points.

[0154] Table 3

[0155] index Corresponding key point index formula Arm circumference (L(p16, p24)+L(p60, p68)) / 2 Thigh circumference (L(p31, p42)+L(p42, p53)) / 2 Calf circumference (L(p34, p39)+L(p45, p50)) / 2

[0156] In the embodiment of the present application, human attributes are divided into two categories. The second category of human attributes includes a chest attribute determined by upper chest circumference and lower chest circumference, a plurality of gears determined by the chest attribute, and a numerical range corresponding to the chest attribute at each gear; a waist attribute, a plurality of gears determined by the waist attribute, and a numerical range corresponding to the waist attribute at each gear; a belly circumference attribute, a plurality of gears determined by the belly circumference attribute, and a numerical range corresponding to the belly circumference attribute at each gear; a hip attribute, a plurality of gears determined by the hip attribute, and a numerical range corresponding to the hip attribute at each gear; a crotch attribute, a plurality of gears determined by the crotch attribute, and a numerical range corresponding to the crotch attribute at each gear. Corresponding numerical ranges; shoulder length attribute, multiple gears determined by shoulder length, and the numerical range corresponding to shoulder length at each gear; arm length attribute, multiple gears determined by arm length, and the numerical range corresponding to arm length at each gear; arm circumference attribute, multiple gears determined by arm circumference, and the numerical range corresponding to arm circumference at each gear; thigh circumference attribute, multiple gears determined by thigh circumference, and the numerical range corresponding to thigh circumference at each gear; calf circumference attribute, multiple gears determined by calf circumference, and the numerical range corresponding to calf circumference at each gear; leg length attribute, multiple gears determined by leg length, and the numerical range corresponding to leg length at each gear.

[0157] Figure 8 Various indicators for evaluating any one of the second category of human attributes are shown. Fig. 9 Another schematic diagram is shown of an indicator for evaluating any one of the second category of human attributes.

[0158] The 11 specific categories of indicators of the human body are determined using the rules defined above. The above rules are summarized based on a preset number of real model data, which can be 1000 or 2000. The following formula is used for calculation:

[0159] C 胸围 =f(upper bust-lower bust); C 腰围 =f(waist / hip);

[0160] C 肚腩围 =f(belly circumference / hip circumference); C 臀围 =f(waist / hip); C 跨围 =f(waist / hip);

[0161] C 肩宽 =f(shoulder length / hip circumference); C 臂围 =f(arm circumference / thigh circumference);

[0162] C 大腿围 =f(thigh circumference / leg length); C 小腿围 =f(calf circumference / leg length);

[0163] C 腿长 =f(leg length / height);

[0164] C 臂长 =f(length from shoulder to base of thumb / vertical length from shoulder to crotch line);

[0165] Among them, C represents the category corresponding to each indicator, and f represents the function that converts the parameters in f into specific categories according to the rules in the above table.

[0166] The embodiment of the present application predicts various human attributes of the user through a first human attribute prediction model and a second human attribute prediction model, wherein the first human attribute prediction model is a prediction model for predicting human attributes based on the user's pupil distance, and the second human attribute prediction model is a preset neural network prediction model.

[0167] After the coordinates of the key points of the user's body contour are identified in step 102, the user's various body attributes are predicted using the first body attribute prediction model and the second body attribute prediction model according to the coordinates of the key points of the user's body contour to obtain a first prediction result and a second prediction result.

[0168] Among them, the first human attribute prediction model predicts the various human attributes of the user through the above formulas (5), (6), (7) and (8) and the methods shown in Tables 1, 2 and 3. Specifically, for any human attribute in the first category of human attributes, determine the calculation formula for calculating the human attribute. According to the coordinates of the key points of the user's human contour and the pupil distance as a reference, predict the human attribute according to the determined calculation formula. Obtain big data that can determine the second category of human attributes; analyze the big data to obtain preset conditions that can determine the second category of human attributes. For any human attribute in the second category of human attributes, obtain a preset sub-condition that matches the human attribute from the preset conditions. Based on the coordinates of the key points of the user's human contour, predict the human attribute according to the preset sub-conditions.

[0169] The second human attribute prediction model is Fig.10 The preset neural network prediction model shown can simultaneously predict 18 specific numerical values ​​(cm) and 11 specific categories (from fine to coarse) of indicators. Based on the preset neural network prediction model, the input is the normalized values ​​corresponding to the coordinates of the user's various human body key points. The normalization method is as follows:

[0170] Assume that the coordinates of the key points of the human body contour are P = {p1, p2, p3, ... p77}. Then, among all the key points of the human body contour of the user, the smallest coordinate upper left corner point Pmin and the largest coordinate lower right corner point Pmax can be determined. Pmin = (min(Px), min(Py)); Pmax = (max(Px), max(Py)). The coordinates of the normalized point P are Pnorm, Pnorm = (P-Pmin) / (Pmax-Pmin).

[0171] like Figure 7 As shown in the figure, there are 77 key points of the user's human body contour, each of which has two coordinate values, x and y. The final network input is 77*2=154. Then it is processed by 5 fully connected layers and outputs 2 branches respectively. The dimension of the first branch is 18, which is mainly responsible for the prediction of 18 specific numerical values ​​(cm). The dimension of the second branch output is 11*5, 11 represents 11 specific categories (from fine to coarse) of indicators, and 5 means that each indicator has 5 categories. In this way, all human attribute indicators are predicted through a neural network.

[0172] It should be noted that the 18 specific numerical values ​​(cm) are normalized output results, and the output result range is also between 0 and 1. Then multiply the output by the user's height to get the final result of each indicator.

[0173] In an embodiment of the present application, the first prediction result obtained by the first human attribute prediction model includes: a prediction result obtained based on pupil distance prediction and used to characterize any human attribute in the first category of human attributes, and a prediction result obtained based on a preset condition and used to characterize any human attribute in the second category of human attributes. The second prediction result obtained by the second human attribute prediction model includes a prediction result obtained based on a preset neural network prediction model and used to characterize any human attribute in the first category of human attributes, and a prediction result obtained based on a preset neural network prediction model and used to characterize any human attribute in the second category of human attributes.

[0174] The first prediction result and the second prediction result are fused to obtain and output a fused prediction result. Specifically, according to the first prediction result, a first weight value preconfigured based on the first prediction result, the second prediction result, and a second weight value preconfigured based on the second prediction result, the first prediction result and the second prediction result are data fused to obtain and output a fused prediction result.

[0175] The embodiment of the present application does not impose any specific restrictions on the specific values ​​of the first weight value and the second weight value. In order to balance the first prediction result and the second prediction result, the first weight value and the second weight value may also be configured to be the same value.

[0176] After predicting the various body attributes of the user in the above manner, a three-dimensional body model of the user is created based on the various body attributes and the coordinates of key points of the body contour.

[0177] In the embodiment of the present application, the range intervals corresponding to each human attribute are pre-set. After predicting each human attribute of the user, the range intervals to which each human attribute belongs are determined respectively according to the specific values ​​of each human attribute and the range intervals corresponding to each human attribute. The attribute type to which each human attribute belongs is determined respectively according to the range interval to which each human attribute belongs. Taking arm length as an example, the range intervals of arm length corresponding to different arm length types are preset in advance. The arm length types can be divided into short arms, medium lengths and long arms. The range intervals corresponding to short arms are AB, the range intervals corresponding to medium lengths are BC, and the range intervals corresponding to long arms are CD. If the user's arm length is within the range interval CD, it is determined that the user's arm length belongs to the long arm type.

[0178] After automatically determining the attribute category to which each of the user's body attributes belongs in the above manner, if the user wants to change his or her attribute category, he or she can also manually click on the attribute category he or she wants to use to modify it.

[0179] According to the coordinates of the key points of the user's body contour, the values ​​of various body attributes and the attribute types of each body attribute, an affine transformation in three-dimensional space is performed to generate a three-dimensional body model of the user. By creating a three-dimensional face model and a three-dimensional body model of the user, a three-dimensional full-body model of the user is obtained.

[0180] Since the user's facial image and body image can be obtained by scanning the user in real time, a facial 3D model is created based on the facial image, and a body 3D model is created based on the body image. The final established full-body 3D model is very consistent with the user's current real situation. Subsequent virtual dressing through the full-body 3D model can bring users a dressing experience that is closer to reality.

[0181] In the embodiment of the present application, after predicting the various human attributes of the user, the user's body analysis data is obtained according to the various human attributes, and the body analysis data includes head-to-body ratio, body mass index, leg shape, etc. After creating the user's full-body three-dimensional model in the above manner, the full-body three-dimensional model and the user's body analysis data are displayed. Figure 3As shown, the key points of the human body contour identified in step B1 are marked in interface (c), and then the creation process of the full-body 3D model is intuitively displayed to the user through interfaces (d) and (e) in turn. Finally, the final full-body 3D model and the user's head-to-body ratio, body mass index, leg shape and other body analysis data are displayed in interface (f).

[0182] In other embodiments of the present application, the user's full body 3D model may not be created based on the user's body image. Instead, an input interface for various body data such as height, weight, shoulder width, arm length, hip circumference, and leg length is provided for the user, and the user manually inputs various body data through these input interfaces. The terminal receives the various body data input by the user and creates a full body 3D model of the user based on the various body data. For example, by Fig.11 The input interface for various body data provided to the user in the middle interface (A) receives the body data input by the user. When the user enters the body data, he can select a data that suits his body from the data options shown in the interface (B). After the user enters all the body data, the full body three-dimensional model shown in the interface (C) is generated.

[0183] In the embodiment of the present application, after creating the full body 3D model of the user, if the user believes that the created full body 3D model does not match his / her real body shape, the full body 3D model can also be adjusted. Specifically, the user can input his / her real body shape data from the body shape data input interface provided by the terminal, and the terminal receives the body shape data input by the user. Based on the body shape data, the full body 3D model of the user is corrected.

[0184] Specifically, the range interval to which the body data belongs is determined based on the body data input by the user and the range interval corresponding to each human attribute. The attribute type to which the body data belongs is determined based on the range interval to which the body data belongs. If the attribute type is different from the attribute type to which the body data belongs in the created full-body three-dimensional model, the part corresponding to the body data in the full-body three-dimensional model is replaced with the attribute type corresponding to the body data. For example, suppose that it is determined that the user belongs to the long arm type based on the arm length input by the user, and the arm length in the previously created full-body three-dimensional model is the medium-length type. Then the arm length in the full-body three-dimensional model is modified to the long arm type.

[0185] In the process of creating the user's full-body 3D model, a user's appearance report is also generated based on the user's facial image. Specifically, based on the coordinates of the user's facial key points, the user's face length, face width, zygomatic arch width, middle and lower face length, middle and lower face ratio and other facial analysis data are calculated.

[0186] The face shape may include a square face, a baby face, a standard face, a long face, etc. The embodiment of the present application pre-sets the face data ranges such as the face length range, the face width range, and the zygomatic arch width range corresponding to each face shape. According to the calculated face analysis data of the user and the face data ranges corresponding to various face shapes, the face data range to which each face analysis data belongs is determined. The face shape corresponding to the determined face data range is obtained, and the face shape is determined as the face shape of the user.

[0187] The user's appearance report includes the above-mentioned facial analysis data such as face length, face width, zygomatic arch width, length of middle and lower face, proportion of middle and lower face, as well as the user's face shape, and includes a summary of the user's appearance advantages and areas for improvement. For example, the appearance advantages may be moderate zygomatic arch width, harmonious proportion of middle and lower face, etc. The areas for improvement may be small eyes, too wide zygomatic arch width, etc.

[0188] The terminal displays the user's appearance report and various face shape options. If the user is not satisfied with the face shape analyzed in the appearance report, he can change it by clicking on a certain face shape option. After the terminal detects that the user clicks on a certain face shape option, it modifies the full body 3D model and the face shape of the user in the appearance report to the face shape selected by the user. Fig.12 In the figure, interface (1) shows the user's appearance report, and interface (2) shows options for facial analysis data such as face length, face width, zygomatic arch width, middle and lower face length, and middle and lower face ratio. Users can submit their own facial data through these options to adjust the full-body three-dimensional model and their own face shape in the appearance report.

[0189] In the embodiment of the present application, in the process of creating a full-body three-dimensional body model of the user, a body report of the user is also generated based on the body image of the user. Specifically, the various human attributes of the user are predicted, and the body mass index, head-to-body ratio and other body proportions of the user are calculated based on the various human attributes of the user. And the body shape of the user is determined based on the various human data and various body proportions of the user. And based on the various human data and various body proportions of the user, the body posture bonus items and items to be improved of the user are determined. For example, the body posture bonus items may be a thin waist, long legs, etc. The items to be improved may be thick calves, beer belly, etc. The generated body report of the user includes the above-mentioned body mass index, head-to-body ratio and other body proportions, the body shape of the user, body posture bonus items and items to be improved, etc. Fig.12 In the interface (four), the user's body report is shown.

[0190] In the process of creating the user's full-body three-dimensional body model, a skin report of the user is also generated based on the user's facial image and body image. The exposed skin part is detected from the facial image and body image through the target detection algorithm. The color, brightness, texture, etc. of the detected skin part are analyzed to obtain the user's current skin condition, skin age, skin nature, spots, acne marks, blackheads, wrinkles, skin color, etc., analyze the user's skin bonus items, and summarize the user's current skin care priorities. Among them, skin bonus items may be water-oil balance, wrinkle-free, shiny, etc. The skin care priorities may be moisturizing, anti-wrinkle, etc. The generated user's skin report includes the above-mentioned skin condition, skin age, skin nature, spots, acne marks, blackheads, wrinkles, skin color, skin bonus items and skin care priorities. Fig.12 In the interface (three), the user's skin report is shown.

[0191] The terminal displays the appearance report, body report and skin report generated above, showing the user's various body data in an all-round way. When viewing these reports, the user can push the report page up to browse the long report; or choose to hide the report part at the bottom and only display the whole body 3D model part.

[0192] After creating a full-body 3D model that meets the actual situation of the user through the method provided in the embodiment of the present application, online fitting can be performed based on the full-body 3D model so that the user can choose the clothes he likes. The embodiment of the present application is pre-configured with a preset clothing library, which includes a large number of clothing parameter information of clothing, shoes, accessories and other clothing. The clothing parameter information includes the name of the clothing, 3D pictures, thumbnails, size data, etc. For example, the clothing parameter information corresponding to a skirt includes the 3D picture, thumbnail and size data of the skirt, such as skirt length, bust, sleeve length, size, etc.

[0193] After creating the user's full-body three-dimensional model, clothing parameter information matching the user can also be obtained based on the user's facial image and body image. Specifically, after obtaining the user's face shape, body shape, chest circumference, waist circumference, hip circumference, head-to-body ratio and other body data based on the user's facial image and body image through the above steps, clothing parameter information matching the user's body is obtained from the preset clothing library based on these body data of the user and the clothing parameter information corresponding to each piece of clothing in the preset clothing library. The matching clothing parameter information is recommended to the user.

[0194] Specifically, the matching degree between the user's body data such as arm length, leg length, chest circumference, waist circumference, hip circumference, height, weight, etc. and the size data such as clothing length, chest circumference, sleeve length, waist circumference, etc. included in the clothing parameter information is calculated, and the clothing parameter information with a matching degree greater than a preset threshold is determined as the clothing parameter information that matches the user. Thumbnails of clothing are obtained from the matching clothing parameter information, and these thumbnails of clothing are displayed to the user so that the user can select the clothing he likes from them for matching and trying on.

[0195] Step 104: Generate a fitting effect diagram based on the full-body three-dimensional model and the clothing parameter information selected by the user.

[0196] The terminal displays the user's full-body 3D model in an interface that includes a fitting interface. When it is detected that the user clicks on the fitting interface, the terminal jumps to the fitting interface, which includes thumbnails of multiple clothing recommended for the user. When the terminal detects that the user clicks on a certain clothing thumbnail, the clothing parameter information corresponding to the thumbnail is obtained. Based on the 3D picture of the clothing and the size data of the clothing included in the clothing parameter information, the clothing is rendered onto the user's full-body 3D model to obtain a fitting effect image corresponding to the clothing.

[0197] In the process of generating the fitting effect diagram, dressing effect information corresponding to the clothing parameter information is generated based on the full body 3D model and clothing parameter information, and the dressing effect information is displayed. That is, the matching degree between the clothing and the user's body and the dressing effect are analyzed based on the full body 3D model and clothing parameter information. While displaying the fitting effect diagram corresponding to the clothing, dressing effect information such as the matching degree, clothing effect, name of the clothing, and size of the clothing can also be displayed.

[0198] like Fig.13 In the interface (five), multiple clothing thumbnails are included. When the user clicks on a thumbnail, the fitting effect picture, clothing matching degree, clothing effect, etc. shown in the interface (six) are displayed.

[0199] The embodiment of the present application identifies the coordinates of the key points of the user's face and body based on the user's full-body image, and creates a real-life full-body three-dimensional model of the user based on this, and can recommend clothing suitable for the user based on the user's real facial features and body features. When the user selects clothing from the recommended clothing for trial wearing, the clothing or accessories are tried on the user's full-body three-dimensional model through the three-dimensional image and size data of the clothing, so that the clothing to be tried on can be consistent with the user's full-body three-dimensional model in terms of height and face, etc., avoiding the problem of clothing skewness and deformation in the generated fitting effect picture, improving the referenceability of the fitting effect picture, and providing users with a virtual fitting experience that is more in line with reality. Furthermore, the embodiment of the present application also generates a face report, a body report, and a skin report for the user based on the user's real facial image and body image, providing accurate data support and professional body management suggestions for the user's body management.

[0200] The present application embodiment provides a virtual dressing device, which is used to execute the virtual dressing method described in the above embodiment. Fig.14 , the device comprises:

[0201] An acquisition module 1401 is used to acquire a full-body image of a user;

[0202] The recognition module 1402 is used to recognize the coordinates of the key points of the user's face and the coordinates of the key points of the body contour from the full-body image;

[0203] A creation module 1403, for creating a full-body three-dimensional model of the user based on the full-body image, the coordinates of the key points of the face, and the coordinates of the key points of the body contour;

[0204] The generation module 1404 is used to generate a fitting effect diagram according to the full body three-dimensional model and the clothing parameter information selected by the user.

[0205] The creation module 1403 is used to create a user's facial 3D model based on the coordinates of the facial key points and the facial image included in the full-body image; create a user's body 3D model based on the coordinates of the human body contour key points to obtain the user's full-body 3D model.

[0206] Creation module 1403 is specifically used to identify the gender category and hairstyle category of the user based on the facial image included in the full-body image; create a three-dimensional facial model of the user based on the gender category, hairstyle category and coordinates of the key points of the face corresponding to the user; and predict the various body attributes of the user based on the coordinates of the key points of the body contour; create a three-dimensional body model of the user based on the various body attributes and the coordinates of the key points of the body contour.

[0207] The device also includes: a clothing recommendation module, which is used to obtain clothing parameter information matching the user based on the full-body image; and recommend the matching clothing parameter information to the user.

[0208] The device also includes: an effect analysis module, which is used to generate dressing effect information corresponding to the clothing parameter information according to the full-body three-dimensional model and the clothing parameter information; and display the dressing effect information.

[0209] The device also includes: a model correction module, which is used to receive body data input by a user; and correct the whole body three-dimensional model according to the body data.

[0210] The device also includes: a report generating module, which is used to generate a user's appearance report based on the facial image; generate a user's body report based on the body image; and generate a user's skin report based on the facial image and the body image.

[0211] The virtual dressing device provided in the above-mentioned embodiment of the present application and the virtual dressing method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0212] The embodiment of the present application also provides an electronic device corresponding to the virtual dressing method provided in the above embodiment, so as to execute the virtual dressing method. The embodiment of the present application is not limited.

[0213] Please refer to Fig.15 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Fig.15 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, and the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and when the processor 200 runs the computer program, it executes the virtual dressing method provided in any of the aforementioned embodiments of the present application.

[0214] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0215] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving the execution instruction. The virtual dressing method disclosed in any implementation of the embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0216] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0217] The electronic device provided in the embodiment of the present application and the virtual dressing method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0218] The present application also provides a computer-readable storage medium corresponding to the virtual dressing method provided in the above embodiment. Fig.16 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the virtual dressing method provided in any of the aforementioned embodiments will be executed.

[0219] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0220] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the virtual dressing method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0221] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0222] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A virtual dressing method, characterized in that: The method comprises: Obtaining a full-body image of the user; Identify the coordinates of the user's facial key points and the coordinates of the user's body contour key points from the full-body image; According to the full-body image, the coordinates of the key points of the face and the coordinates of the key points of the body contour, a full-body three-dimensional model of the user is created; including: according to the coordinates of the key points of the face and the face image included in the full-body image, a three-dimensional face model of the user is created; according to the coordinates of the key points of the body contour, a three-dimensional body model of the user is created, including: according to the coordinates of the key points of the body contour, various human attributes of the user are predicted; various human attributes of the user are predicted by a first human attribute prediction model and a second human attribute prediction model, the first human attribute prediction model obtains a first prediction result, the second human attribute prediction model obtains a second prediction result, the first prediction result and the second prediction result are fused to obtain and output the fused prediction result; wherein the first human attribute prediction model is a prediction model for predicting human attributes based on the user's pupil distance, and the second human attribute prediction model is a preset neural network prediction model; according to the various human attributes and the coordinates of the key points of the body contour, a three-dimensional body model of the user is created; A fitting effect diagram is generated according to the full-body three-dimensional model and the clothing parameter information selected by the user.

2. The method according to claim 1, characterized in that: The step of creating a three-dimensional facial model of the user according to the coordinates of the facial key points and the facial image included in the full-body image comprises: According to the facial image included in the full-body image, identifying the gender category and hairstyle category of the user; A three-dimensional facial model of the user is created according to the gender category, the hairstyle category and the coordinates of the facial key points corresponding to the user.

3. The method according to claim 1, characterized in that: Before generating the fitting effect diagram according to the full-body three-dimensional model and the clothing parameter information selected by the user, the method further includes: Acquiring clothing parameter information matching the user according to the full-body image; The matching clothing parameter information is recommended to the user.

4. The method according to claim 1, characterized in that After generating a fitting effect diagram according to the full-body three-dimensional model and the clothing parameter information selected by the user, the method further includes: Generating dressing effect information corresponding to the clothing parameter information according to the full-body three-dimensional model and the clothing parameter information; Display the costume effect information.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Receive body data input by the user; The whole body three-dimensional model is modified according to the body shape data.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: generating an appearance report of the user according to the facial image included in the full-body image; generating a body report of the user according to the body image included in the full-body image; A skin report of the user is generated according to the facial image and the body image.

7. A virtual dressing device, characterized in that: The device comprises: An acquisition module, used to acquire a full-body image of a user; A recognition module, used to recognize the coordinates of the key points of the face and the key points of the body contour of the user from the full-body image; A creation module, for creating a full-body three-dimensional model of the user according to the full-body image, the coordinates of the facial key points and the coordinates of the human body contour key points; including: creating a three-dimensional facial model of the user according to the coordinates of the facial key points and the facial image included in the full-body image; creating a three-dimensional body model of the user according to the coordinates of the human body contour key points, including: predicting various human attributes of the user according to the coordinates of the human body contour key points; predicting various human attributes of the user by a first human attribute prediction model and a second human attribute prediction model, the first human attribute prediction model obtaining a first prediction result, the second human attribute prediction model obtaining a second prediction result, fusing the first prediction result and the second prediction result to obtain and output the fused prediction result; wherein the first human attribute prediction model is a prediction model for predicting human attributes based on the user's pupil distance, and the second human attribute prediction model is a preset neural network prediction model; creating a three-dimensional body model of the user according to the various human attributes and the coordinates of the human body contour key points; A generation module is used to generate a fitting effect diagram according to the full-body three-dimensional model and the clothing parameter information selected by the user.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images