Image processing method, device, electronic device and readable storage medium
By determining whether the portrait is in a mouth-open state and using a pre-trained tooth beautification model, the teeth in the image are beautified, which solves the problem that the prior art cannot beautify teeth and improves the user experience.
Patent Information
- Application Number
- CN202011455576.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-12-10
AI Technical Summary
Existing beauty treatment technology cannot beautify teeth, resulting in the problem of imperfect teeth in the image.
By obtaining the image to be processed, we judge whether the portrait is in the mouth-open state, and input it into the pre-trained tooth beautification model for tooth beautification. We use the neural network and the preset training data set to calculate the loss function, and update the neural network parameters until the preset is reached Threshold, generates a result image of the teeth beautification.
The beautification of teeth in the image is realized, the user experience is improved, and the problem that the existing technology cannot beautify teeth is solved.
Smart Images

Figure CN112561816B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art
[0002] Since its popularity, beauty filters have become a way of presenting in the national social network. It beautifies images through face detection, feature point positioning, special effects processing, skin smoothing and whitening, etc. Beauty technology has built a dream stage for people in online social networking to decorate themselves, and people can show themselves to their heart's content on such a stage.
[0003] Current beautification processing only performs beautification adjustments on the face or torso. For example, skin smoothing, face slimming and other beautification operations are performed on portraits, but there is a problem that teeth cannot be beautified. Summary of the invention
[0004] In view of this, the present application provides an image processing method, device, electronic device and readable storage medium to improve the above-mentioned problems.
[0005] In a first aspect, the present invention provides an image processing method, applied to an electronic device, the method comprising:
[0006] Acquire an image to be processed, wherein the image to be processed includes a portrait;
[0007] It is determined whether the person in the image to be processed is in an open mouth state. If so, the image to be processed is input into a pre-trained teeth beautification model for teeth beautification to obtain a teeth beautification result image.
[0008] In an optional embodiment, the electronic device stores a pre-constructed training data set and a pre-constructed neural network, the training data set includes a plurality of tooth images, and the method further includes a step of obtaining an image processing model, the step comprising:
[0009] Acquire an original image and a first target tooth image, wherein the original image includes a portrait of a person showing teeth, the original image and the first target tooth image are arbitrarily selected from the training data set, and the original image is different from the first target tooth image;
[0010] The original image and the first target tooth image are used as training samples and input into the neural network to obtain a preliminary result image after beautifying the teeth of the portrait in the original image;
[0011] Calculating a loss value of a preset loss function using the preliminary result image and the original image;
[0012] Determine whether the loss value is less than a first preset threshold value. If so, use the neural network as a tooth beautification model. If not, use a back propagation algorithm to update the parameters of the neural network, and execute the step of acquiring the original image and the first target tooth image again to determine whether the loss value is less than the first preset threshold value, until the recalculated output value is less than the first preset threshold value, and obtain a tooth beautification model.
[0013] In an optional implementation, the loss function includes an L1 loss function, an L2 loss function, a perceptual loss function, and a generative adversarial loss function; and the step of calculating the loss value of a preset loss function using the preliminary result image and the original image includes:
[0014] Calculating a first output value of the L1 loss function according to the preliminary result image and the original image;
[0015] Calculating a second output value of the L2 loss function according to the preliminary result image and the original image;
[0016] Calculating a third output value of the perceptual loss function according to the preliminary result image and the original image;
[0017] Calculating a fourth output value of the generative adversarial loss function according to the preliminary result image and the original image;
[0018] A sum of the first output value, the second output value, the third output value, and the fourth output value is calculated, and the sum is used as the loss value.
[0019] In an optional embodiment, the electronic device further stores a pre-trained perception model, wherein the perception model includes a plurality of perception layers;
[0020] The step of calculating the third output value of the perceptual loss function according to the preliminary result image and the original image comprises:
[0021] Inputting the preliminary result image into the perception model to obtain a first feature map output by each perception layer;
[0022] Inputting the original image into the perception model to obtain a second feature map output by each perception layer;
[0023] According to all the first feature maps and the second feature maps, a third output value of the perceptual loss function is calculated according to the following formula:
[0024]
[0025] Among them, L Vgg is the perceptual loss function, is the first feature map output by the i-th perception layer, It is the second feature map output by the i-th perception layer.
[0026] In an optional embodiment, the neural network includes a discriminant network, and the step of calculating the fourth output value of the generative adversarial loss function according to the preliminary result image and the original image includes:
[0027] According to the preliminary result image and the original image, a fourth output value of the generative adversarial loss function is calculated according to the following formula:
[0028]
[0029] Among them, L adv is the generative adversarial loss function, x is the original image, P x is the distribution of multiple tooth images in the training dataset, P z is the distribution of the preliminary result image, D(x) is the output result after the original image is input into the discriminant network, D(z) is the output result after the preliminary result image is input into the discriminant network, and E represents expectation.
[0030] In an optional implementation, the step of determining whether the human figure in the image to be processed is in an open mouth state includes:
[0031] Acquire a target area image from the image to be processed, wherein the target area image includes a mouth area of the portrait;
[0032] Marking feature points of the mouth area of the target area image to obtain a plurality of feature points;
[0033] Calculating the distances between the feature points at preset locations among the feature points;
[0034] It is determined whether the distance is greater than a second preset threshold value. If so, it is determined that the portrait in the image to be processed is in an open-mouth state. If not, it is determined that the portrait in the image to be processed is not in an open-mouth state.
[0035] In an optional embodiment, the electronic device stores a pre-constructed training data set, and the training data set includes a plurality of tooth images;
[0036] The step of inputting the image to be processed into a pre-trained teeth beautification model to perform teeth beautification to obtain a teeth beautification result image comprises:
[0037] Preprocessing the image to be processed to obtain a preprocessing result image;
[0038] Acquire a second target tooth image, wherein the second target tooth image is arbitrarily selected from the training data set;
[0039] The second target tooth image and the preprocessing result image are input into the tooth beautification model to perform tooth beautification, and a tooth beautification result image of the teeth of the portrait in the image to be processed is obtained.
[0040] In a second aspect, the present invention provides an image processing device, applied to an electronic device, the device comprising:
[0041] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a portrait;
[0042] The teeth beautification module is used to determine whether the person in the image to be processed is in an open mouth state. If so, the image to be processed is input into a pre-trained teeth beautification model for teeth beautification to obtain a teeth beautification result image.
[0043] In a third aspect, the present invention provides an electronic device, comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the image processing method described in any one of the aforementioned embodiments.
[0044] In a fourth aspect, the present invention provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed, the steps of the image processing method described in any one of the aforementioned embodiments are implemented.
[0045] The embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, by acquiring an image to be processed, wherein the image to be processed includes a portrait, and then determining whether the portrait in the image to be processed is in an open-mouth state. If so, the image to be processed is input into a pre-trained teeth beautification model for teeth beautification to obtain a teeth beautification result image. Without other operations, the imperfect teeth image in the image can be processed for teeth beautification, thereby improving the user experience.
[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and understandable, some examples are given below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 A structural block diagram of an electronic device provided in an embodiment of the present application.
[0049] Figure 2 A face annotation schematic diagram is also provided in an embodiment of the present application.
[0050] Figure 3 This is one of the flowcharts of an image processing method provided in an embodiment of the present application.
[0051] Figure 4 This is a second flowchart of an image processing method provided in an embodiment of the present application.
[0052] Figure 5 A tooth mask image provided in an embodiment of the present application.
[0053] Figure 6 A lip mean filling image provided in an embodiment of the present application.
[0054] Figure 7 A lip zeroing image provided in an embodiment of the present application.
[0055] Figure 8 A schematic diagram showing the comparison of images before and after processing provided in an embodiment of the present application.
[0056] Fig. 9 A functional module block diagram of an image processing device also provided in an embodiment of the present application.
[0057] Icons: 100 - electronic device; 110 - memory; 120 - processor; 130 - image processing device; 131 - acquisition module; 132 - teeth beautification module; 140 - communication unit. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0059] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0060] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0061] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.
[0062] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.
[0063] As introduced in the background technology, beauty filters have become a way of presenting in the national social network since they became popular in 2013. They beautify images through face detection, feature point positioning, special effects processing, skin smoothing and whitening, etc. Beauty technology has built a dream stage for people in online social networking to decorate themselves, and people can show themselves to their heart's content on such a stage.
[0064] Current beautification processing only performs beautification adjustments on the face or torso. For example, skin smoothing, face slimming and other beautification operations are performed on portraits, but there is a problem that teeth cannot be beautified.
[0065] In view of this, the embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which use a pre-trained teeth beautification model to correct and beautify the teeth of a portrait with an open mouth.
[0066] The defects existing in the solutions in the above-mentioned prior art are the results obtained by the applicant after practice and careful research. Therefore, the process of discovering the above-mentioned problems and the solutions proposed in the embodiments of the present application below for the above-mentioned problems should be the contributions made by the applicant to the present application during the application process.
[0067] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and key points in the embodiments can be combined with each other.
[0068] Please refer to Figure 1 , Figure 1This is a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The device may include a processor 120, a memory 110, an image processing device 130, and a communication unit 140. The memory 110 stores machine-readable instructions executable by the processor 120. When the electronic device 100 is running, the processor 120 and the memory 110 communicate through a bus, and the processor 120 executes the machine-readable instructions and performs the image processing method.
[0069] The memory 110 , the processor 120 , and the communication unit 140 are electrically connected to each other directly or indirectly to achieve signal transmission or interaction.
[0070] For example, these components may be electrically connected to each other via one or more communication buses or signal lines. The image processing device 130 includes at least one software function module that may be stored in the memory 110 in the form of software or firmware. The processor 120 is used to execute the executable module stored in the memory 110, such as the software function module or computer program included in the image processing device 130.
[0071] Among them, the memory 110 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0072] The processor 120 may be an integrated circuit chip with signal processing capability. The processor 120 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like.
[0073] It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0074] In the embodiment of the present application, the memory 110 is used to store the program, and the processor 120 is used to execute the program after receiving the execution instruction. The process definition method disclosed in any implementation of the embodiment of the present application can be applied to the processor 120 or implemented by the processor 120.
[0075] The communication unit 140 is used to establish a communication connection between the electronic device 100 and other electronic devices through a network, and to send and receive data through the network.
[0076] In some embodiments, the network may be any type of wired or wireless network, or a combination thereof. By way of example only, the network may include a wired network, a wireless network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof.
[0077] In the embodiment of the present application, the electronic device 100 may be, but is not limited to, a smart phone, a personal computer, a tablet computer, or other device with processing capabilities.
[0078] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also have Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown. Figure 1 The components shown may be implemented in hardware, software or a combination thereof.
[0079] The following is based on Figure 1 The structural diagram of the electronic device 100 illustrates in detail the steps of the image processing method provided in the embodiment of the present application.
[0080] In an optional embodiment, the electronic device 100 stores a pre-constructed training data set and a pre-constructed neural network, and the training data set includes multiple teeth images. At the same time, the teeth images in the training data set can be obtained by cropping the mouth area from a large number of beautiful portrait images with different postures, different mouth openings, and feature points are marked. For example, the training data set can be obtained by collecting facial images with different angles and postures and different mouth openings, such as: front face, side face, head up, only upper teeth exposed, and both upper and lower teeth exposed. The mouth area is cropped out using facial points and rotated to horizontal. At the same time, multiple gains are added to the data, such as flipping, color cast, rotation, translation, affine transformation, exposure, contrast adjustment, blurring, etc. The processed image is as follows: Figure 2 As shown in the figure, face points 0-11 are the outer circle points of the mouth, and face points 12-19 are the inner circle points of the mouth.
[0081] Optionally, see Figure 3 , the neural network can be trained using the training data set to obtain a trained teeth beautification model by the following method:
[0082] Step S100, acquiring an original image and a first target tooth image.
[0083] The original image includes a portrait of a person showing teeth, the original image and the first target tooth image are arbitrarily selected from a training data set, and the original image is different from the first target tooth image.
[0084] Step S200, the original image and the first target tooth image are used as training samples and input into a neural network to obtain a preliminary result image after beautifying the teeth of the portrait in the original image.
[0085] Step S300, using the preliminary result image and the original image, calculate the loss value of the preset loss function.
[0086] Step S400, determining whether the loss value is less than a first preset threshold value. If yes, executing step S500. If no, executing step S600.
[0087] Step S500, using the neural network as a teeth beautification model.
[0088] Step S600, using a back propagation algorithm to update the parameters of the neural network.
[0089] Steps S100 to S400 are executed again until the recalculated loss value is less than the first preset threshold value, thereby obtaining a teeth beautification model.
[0090] Among them, the pre-built neural network model can be shown in the following table:
[0091] Table 1
[0092]
[0093]
[0094]
[0095]
[0096] The neural network includes a first branch, a second branch and a main network, each branch includes an input layer, multiple convolution activation layers, and multiple normalization layers. As shown in Table 1, the first branch includes branch_1_input layer to GroupNorm_3 layer. The second branch includes branch_2_input to GroupNorm_6 layer. The main network includes concat layer to Output layer.
[0097] Layer represents a layer in the structure, ConvX_LeakyReLU represents that a ReLU activation operation will be performed after a convolution operation is performed on convolution layer X, GroupNorm_X represents that a normalization operation will be performed after a convolution and activation operation is performed on convolution layer X, ResBlock_X represents the Xth residual block, and ConvX represents that a convolution operation is performed only on convolution layer X. Concat represents the merging of two or more arrays. For example, in an embodiment of the present application, the output value of the first branch can be merged with the output value of the second branch. Upsampe_X represents the Xth upsampling layer. Tanh represents the activation layer, and Output represents the output layer.
[0098] K represents Kernel, which is the convolution kernel, kh represents the height of the convolution kernel, kw represents the width of the convolution kernel, padding is the padding parameter, stride is the step size of the convolution kernel, and Scale_factor is the upsampling scale factor. Group is, Mode = 'bilinear' means that the 'bilinear' upsampling algorithm is used for upsampling. Optionally, other upsampling algorithms can also be used for upsampling, such as 'nearest', 'linear', 'bicubic' and 'trilinear'.
[0099] Thus, in combination with the structure of the above neural network, the neural network can be trained by the method shown in step S100 to step S600 to obtain a trained teeth beautification model.
[0100] For example, the original image is input into the input layer of the first branch, the first target tooth image is input into the input layer of the second branch, and the first branch and the second branch are used to extract features of the image respectively. Then, the features extracted by the first branch and the second branch are merged, and the main network is used to decode the features to obtain a preliminary result image. The loss value of the preset loss function is calculated using the preliminary result image and the original image until the loss value is less than the first preset threshold, and a tooth beautification model is obtained.
[0101] It is understandable that the process of training the neural network to obtain the teeth beautification model can be performed in other electronic devices, and after the training is completed, the teeth beautification model can be transplanted into the electronic device 100. Optionally, the process of training the neural network to obtain the teeth beautification model can also be performed in the electronic device 100, and after the training is completed, it can be saved.
[0102] In an optional implementation, the loss function includes an L1 loss function, an L2 loss function, a perceptual loss function, and a generative adversarial loss function; step S200 may be implemented in the following manner:
[0103] According to the preliminary result image and the original image, the first output value of the L1 loss function is calculated.
[0104] According to the preliminary result image and the original image, the second output value of the L2 loss function is calculated.
[0105] According to the preliminary result image and the original image, a third output value of the perceptual loss function is calculated.
[0106] Based on the preliminary result image and the original image, a fourth output value of the generated adversarial loss function is calculated.
[0107] The sum of the first output value, the second output value, the third output value, and the fourth output value is calculated, and the sum is used as the loss value.
[0108] That is to say, the loss value of the loss function in the embodiment of the present application can be realized by the following formula:
[0109] L=L1+L2+L vgg +L adv
[0110] Wherein, L is the above loss function, and its output is the loss value. L1 is the L1 loss function, and its output is the first output value. L2 is the L2 loss function, and its output is the second output value. vgg is the perceptual loss function, and its output value is the third output value. adv To generate an adversarial loss function, its output value is the fourth output value.
[0111] Through this loss function, it can be ensured that the tooth information in the preliminary result image obtained after teeth beautification is complete and clear.
[0112] In an optional implementation, the electronic device further stores a pre-trained perception model, where the perception model includes multiple perception layers. Optionally, the perception model may be a VGG model.
[0113] The third output value of the perceptual loss function can be calculated based on the preliminary result image and the original image by the following steps:
[0114] The step of calculating the third output value of the perceptual loss function according to the preliminary result image and the original image includes:
[0115] Input the preliminary result image into the perception model to obtain the first feature map output by each perception layer;
[0116] Input the original image into the perception model to obtain the second feature map output by each perception layer;
[0117] According to all the first feature maps and the second feature maps, the third output value of the perceptual loss function is calculated according to the following formula:
[0118]
[0119] Where i = [0, N], N is a positive integer, i represents the i-th perception layer included in the perception model, L Vgg is the perceptual loss function, is the first feature map output by the i-th perception layer, It is the second feature map output by the i-th perception layer.
[0120] Through this perceptual loss function, the information of the non-lip area in the preliminary result image can be supervised to ensure that the non-lip area does not change and the consistency of facial features.
[0121] In an optional embodiment, the neural network includes a discriminant network, and the step of calculating the fourth output value of the adversarial loss function according to the preliminary result image and the original image includes:
[0122] According to the preliminary result image and the first target tooth image, the fourth output value of the generated adversarial loss function is calculated according to the following formula:
[0123]
[0124] Among them, L adv is the generative adversarial loss function, x is the original image, P x is the distribution of multiple tooth images in the training dataset, P zis the distribution of the preliminary result image, D(x) is the output result after the original image is input into the discriminant network, D(z) is the output result after the preliminary result image is input into the discriminant network, and E represents expectation.
[0125] Please refer to Figure 4 , Figure 4 The second flowchart of the image processing method provided in the embodiment of the present application includes the following steps:
[0126] Step S1, obtaining an image to be processed, wherein the image to be processed includes a portrait.
[0127] Step S2, determining whether the person in the image to be processed is in an open mouth state. If yes, proceed to step S3.
[0128] Step S3, inputting the image to be processed into a pre-trained teeth beautification model to perform teeth beautification, and obtaining a teeth beautification result image.
[0129] The image to be processed may be acquired by a camera included in the electronic device, or may be an image publicly available on the Internet.
[0130] Meanwhile, if the person in the image to be processed is not in an open-mouth state, other image processing may be performed on the image to be processed, such as beautification, etc. Optionally, step S1 may be executed again to obtain a new image to be processed, again without limitation.
[0131] In an optional implementation manner, step S2, determining whether the human figure in the image to be processed is in an open mouth state, can be implemented in the following manner:
[0132] A target area image is obtained from the image to be processed, wherein the target area image includes a mouth area of the portrait.
[0133] The mouth area of the target area image is annotated with feature points to obtain multiple feature points.
[0134] Calculate the distance between the feature points at the preset locations in the feature points.
[0135] It is determined whether the distance is greater than a second preset threshold value. If so, it is determined that the portrait in the image to be processed is in an open-mouth state. If not, it is determined that the portrait in the image to be processed is not in an open-mouth state.
[0136] Please refer again Figure 2 As an optional implementation, the image to be processed can be processed by image cropping and other processing to obtain a target area image including the mouth area of the portrait. Then, the mouth area of the target area image is annotated with feature points to obtain multiple feature points, such as Figure 2In the figure, feature points 0-11 are points outside the mouth, and face points 12-19 are points inside the mouth.
[0137] As can be seen from the figure, when the mouth is in an open state, some feature point pairs will be at a certain distance, and when the mouth is in a closed state, these feature point pairs will overlap. For example, feature point 13 and feature point 19 in a feature point pair, feature point 14 and feature point 18 in a feature point pair, and feature point 15 and feature point 17 in a feature point pair. In this way, a feature point pair can be arbitrarily selected and the distance between its feature points in the image can be calculated to determine whether the portrait in the image to be processed is in an open mouth state.
[0138] Optionally, the distance may be a Euclidean distance, and the second preset threshold may be set according to actual needs, such as 1, 2, etc.
[0139] In this way, before performing teeth beautification on the image to be processed, it is first determined whether the person in the image is in an open-mouth state. After determining that the person is in an open-mouth state, teeth beautification is performed to avoid a sense of disharmony in the beautified image.
[0140] In an optional embodiment, the electronic device stores a pre-built training data set, and the training data set includes a plurality of tooth images.
[0141] Step S3, inputting the image to be processed into a pre-trained teeth beautification model for teeth beautification, and obtaining the teeth beautification result image can be achieved by the following methods:
[0142] The image to be processed is preprocessed to obtain a preprocessing result image.
[0143] A second target tooth image is obtained, wherein the second target tooth image is arbitrarily selected from the training data set.
[0144] The second target tooth image and the preprocessing result image are input into the tooth beautification model for tooth beautification, and a tooth beautification result image is obtained after the teeth of the portrait in the processed image are beautified.
[0145] As an optional implementation, the image to be processed may be preprocessed in the following manner:
[0146] Likewise Figure 2 As an example, the image to be processed is first cropped to obtain an image including the mouth area, and then the image including only the mouth area is annotated with feature points (i.e., Figure 2 As shown), feature points 0-11 are the outer circle points of the mouth, and face points 12-19 are the inner circle points of the mouth, and are rotated to be horizontal.
[0147] Then as Figure 5As shown in , the mask image of the tooth area can be calculated based on the inner circle points. Then, the color mean of the lip area and the color mean of the tooth area are calculated based on the inner circle points and the outer circle points. The calculated color mean of the lip area is filled into the lip area, and the color mean of the tooth area is filled into the tooth area to obtain the lip mean filled image. The obtained image is as shown in Figure 6 Next, we can set all color values except the mouth area to zero based on the outer circle points to obtain a lip zero image. The resulting image is as follows: Figure 7 shown.
[0148] Finally, the lip mean filled image and the lip zeroed image are merged. Figure 6 and Figure 7 The merging process is performed to complete the preprocessing and obtain the preprocessing result image.
[0149] The preprocessing result image and the second target image randomly selected from the training data set are input into the teeth beautification model, and the teeth beautification processing is performed to obtain the teeth beautification result image in which the second target teeth image is replaced with the image to be processed. Figure 8 As shown, Figure 8 The image on the left is the image to be processed (the image has been mosaicked. It should be noted that the image is only for illustration and will not be mosaicked in actual applications). It can be seen that the teeth of the person in the image are uneven and not beautiful enough. Figure 8 The picture on the right is the Figure 3 The teeth beautification image obtained by the image processing method shown in the figure shows that the teeth of the person in the teeth beautification result image are neat and beautiful.
[0150] Based on the same invention concept, please refer to Fig. 9 , Fig. 9 This is a functional module block diagram of the image processing device provided in the embodiment of the present application. Figure 3 The image processing method shown corresponds to an image processing device 130, the device comprising:
[0151] The acquisition module 131 is used to acquire the image to be processed, wherein the image to be processed includes a portrait.
[0152] The teeth beautification module 132 is used to determine whether the person in the image to be processed is in an open mouth state. If so, the image to be processed is input into a pre-trained teeth beautification model for teeth beautification to obtain a teeth beautification result image.
[0153] Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned image processing method, device, electronic device and readable storage medium in the embodiment of the present application, the implementation principle of the device can refer to the implementation principle of the method, and the repeated parts will not be repeated.
[0154] An embodiment of the present application also provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the above-mentioned image processing method, device, electronic device and readable storage medium are implemented.
[0155] In summary, the embodiments of the present application provide an image processing method, device, electronic device 100 and readable storage medium, by obtaining an image to be processed, wherein the image to be processed includes a portrait, and then determining whether the portrait in the image to be processed is in an open-mouth state. If so, the image to be processed is input into a pre-trained teeth beautification model for teeth beautification to obtain a teeth beautification result image. Without other operations, the imperfect teeth image in the image can be processed for teeth beautification, thereby improving the user experience.
[0156] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. An image processing method, characterized in that: Applied to an electronic device, the electronic device stores a plurality of tooth images and a pre-trained perception model; the method comprises: Acquire an image to be processed, wherein the image to be processed includes a portrait; Determine whether the portrait in the image to be processed is in an open-mouth state. If so, preprocess the image to be processed to obtain a preprocessed image result, and obtain a second target tooth image from the multiple tooth images. Input the second target tooth image and the preprocessed image result into a pre-trained tooth beautification model for tooth beautification to obtain a tooth beautification result image in which the second target tooth image is replaced with the teeth in the image to be processed, wherein the teeth in the tooth beautification result image are neater than the teeth in the image to be processed. The perception model is used to calculate the output value of the loss function related to the neural network model according to the preliminary result image output by the neural network corresponding to the tooth beautification model and the original image input to the neural network model during the training process of the tooth beautification model, so as to supervise whether the information of the non-lip area in the preliminary result image changes; The step of preprocessing the image to be processed to obtain a preprocessed image result includes: Cropping the image to be processed to obtain an image including a mouth area; Based on the image of the mouth area, the inner circle points and the outer circle points of the mouth are annotated; Based on the inner and outer points of the mouth, the color mean of the lip area and the color mean of the tooth area are calculated respectively; Filling the lip region with the color mean value, and filling the tooth region with the color mean value to obtain a lip mean-filled image; Based on the outer circle points of the mouth, all color values of the area except the mouth are set to zero to obtain a lip zero image; The lip mean filling image and the lip zeroing image are merged to obtain the preprocessed image result.
2. The image processing method according to claim 1, characterized in that: The electronic device stores a pre-constructed training data set and a pre-constructed neural network, wherein the training data set includes a plurality of tooth images, and the method further includes a step of obtaining an image processing model, which step includes: Acquire an original image and a first target tooth image, wherein the original image includes a portrait of a person showing teeth, the original image and the first target tooth image are arbitrarily selected from the training data set, and the original image is different from the first target tooth image; The original image and the first target tooth image are used as training samples and input into the neural network to obtain a preliminary result image after beautifying the teeth of the portrait in the original image; Calculating a loss value of a preset loss function using the preliminary result image and the original image; Determine whether the loss value is less than a first preset threshold value. If so, use the neural network as a tooth beautification model. If not, use a back propagation algorithm to update the parameters of the neural network, and execute the step of acquiring the original image and the first target tooth image again to determine whether the loss value is less than the first preset threshold value, until the recalculated output value is less than the first preset threshold value, and obtain a tooth beautification model.
3. The image processing method according to claim 2, characterized in that: The loss function includes an L1 loss function, an L2 loss function, a perceptual loss function, and a generative adversarial loss function; the step of calculating the loss value of the preset loss function using the preliminary result image and the original image includes: Calculating a first output value of the L1 loss function according to the preliminary result image and the original image; Calculating a second output value of the L2 loss function according to the preliminary result image and the original image; Calculating a third output value of the perceptual loss function according to the preliminary result image and the original image; Calculating a fourth output value of the generative adversarial loss function according to the preliminary result image and the original image; A sum of the first output value, the second output value, the third output value, and the fourth output value is calculated, and the sum is used as the loss value.
4. The image processing method according to claim 3, characterized in that: The perception model includes multiple perception layers; The step of calculating the third output value of the perceptual loss function according to the preliminary result image and the original image comprises: Inputting the preliminary result image into the perception model to obtain a first feature map output by each perception layer; Inputting the original image into the perception model to obtain a second feature map output by each perception layer; According to all the first feature maps and the second feature maps, a third output value of the perceptual loss function is calculated according to the following formula: Among them, L Vgg is the perceptual loss function, is the first feature map output by the i-th perception layer, It is the second feature map output by the i-th perception layer.
5. The image processing method according to claim 3, characterized in that: The neural network includes a discriminant network, and the step of calculating the fourth output value of the generative adversarial loss function according to the preliminary result image and the original image includes: According to the preliminary result image and the original image, a fourth output value of the generative adversarial loss function is calculated according to the following formula: Among them, L adv is the generative adversarial loss function, x is the original image, P x is the distribution of multiple tooth images in the training dataset, P z is the distribution of the preliminary result image, D(x) is the output result after the original image is input into the discriminant network, D(z) is the output result after the preliminary result image is input into the discriminant network, and E represents expectation.
6. The image processing method according to claim 1, characterized in that: The step of determining whether the human figure in the image to be processed is in an open mouth state comprises: Acquire a target area image from the image to be processed, wherein the target area image includes a mouth area of the portrait; Marking feature points of the mouth area of the target area image to obtain a plurality of feature points; Calculating the distances between the feature points at preset locations among the feature points; It is determined whether the distance is greater than a second preset threshold value. If so, it is determined that the portrait in the image to be processed is in an open-mouth state. If not, it is determined that the portrait in the image to be processed is not in an open-mouth state.
7. An image processing device, characterized in that: Applied to an electronic device, the electronic device stores a plurality of tooth images and a pre-trained perception model; the device comprises: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a portrait; A teeth beautification module, used to determine whether the portrait in the image to be processed is in an open-mouthed state, and if so, preprocess the image to be processed to obtain a preprocessed image result, and obtain a second target tooth image from the multiple tooth images, and input the second target tooth image and the preprocessed image result into a pre-trained tooth beautification model for tooth beautification to obtain a tooth beautification result image in which the second target tooth image is replaced with the teeth in the image to be processed, wherein the teeth in the tooth beautification result image are neater than the teeth in the image to be processed, and the perception model is used to calculate the output value of the loss function related to the neural network model according to the preliminary result image output by the neural network corresponding to the tooth beautification model and the original image input to the neural network model during the training process of the tooth beautification model, so as to supervise whether the information of the non-lip area in the preliminary result image changes; The step of preprocessing the image to be processed by the teeth beautification module to obtain the preprocessed image result includes: Cropping the image to be processed to obtain an image including a mouth area; Based on the image of the mouth area, the inner circle points and the outer circle points of the mouth are annotated; Based on the inner and outer points of the mouth, the color mean of the lip area and the color mean of the tooth area are calculated respectively; Filling the lip region with the color mean value, and filling the tooth region with the color mean value to obtain a lip mean-filled image; Based on the outer circle points of the mouth, all color values of the area except the mouth are set to zero to obtain a lip zero image; The lip mean filling image and the lip zeroing image are merged to obtain the preprocessed image result.
8. An electronic device, characterized in that: The electronic device includes a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of the image processing method described in any one of claims 1-6.
9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the steps of the image processing method according to any one of claims 1 to 6 are implemented.
