Image processing model training method, image processing method, device and medium

Abstract

The present disclosure provides a training method for an image processing model, an image processing method, a device and a medium. It relates to the field of computer technology, and specifically to the field of artificial intelligence technology such as computer vision, deep learning, and augmented reality. The specific implementation scheme is: input a sample facial image into a first image processing model to obtain the texture coefficient of the sample facial image; generate an initial texture map based on the texture coefficient and the texture base; input a target area map extracted from the sample facial image into a second image processing model to obtain a target object offset feature map of the sample facial image; obtain a final texture map based on the initial texture map and the target object offset feature map; generate a rendering of the sample facial image based on the final texture map; construct a loss function based on the rendering map and the label rendering map; and adjust the parameters of the first image processing model and the second image processing model respectively based on the loss function. According to the technical solution of the present disclosure, the similarity between the reconstructed result and the original face can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and specifically to the field of artificial intelligence technologies such as computer vision, deep learning, and augmented reality. Background Art

[0002] With the rapid development of computer vision, computer technology and other related fields, facial 3D reconstruction technology is also constantly updating. In addition to reconstructing the facial shape, facial 3D reconstruction also requires generating the corresponding facial texture map. How to improve the similarity between the reconstruction result and the original face is a topic that needs to be studied and improved urgently. Summary of the invention

[0003] The present disclosure provides an image processing model training method, an image processing method, an apparatus and a medium.

[0004] According to a first aspect of the present disclosure, a method for training an image processing model is provided, comprising:

[0005] Inputting the sample facial image into the first image processing model, and obtaining a texture coefficient of the sample facial image output by the first image processing model;

[0006] generating an initial texture map of the sample face image based on the texture coefficient and the texture basis;

[0007] Inputting the target area map extracted from the sample facial image into the second image processing model, and obtaining the target object offset feature map of the sample facial image output by the second image processing model;

[0008] Based on the initial texture map and the target object offset feature map, a final texture map is obtained;

[0009] Generate a rendering of the sample face image based on the final texture image;

[0010] Construct a loss function based on the rendered image and the labeled rendered image of the sample face image;

[0011] The parameters of the first image processing model and the second image processing model are adjusted respectively based on the loss function.

[0012] According to a second aspect of the present disclosure, there is provided an image processing method, comprising:

[0013] Inputting the facial image to be processed into the first image processing model, and obtaining the texture coefficient of the facial image to be processed output by the first image processing model;

[0014] Generate an initial texture map of the face image to be processed based on the texture coefficient and the texture basis;

[0015] Inputting the target area map extracted from the face image to be processed into the second image processing model, and obtaining the target object offset feature map of the face image to be processed output by the second image processing model;

[0016] Based on the initial texture map and the target object offset feature map, a final texture map is obtained;

[0017] Among them, the first image processing model and the second image processing model are obtained by using the training method of the image processing model provided according to the first aspect.

[0018] According to a third aspect of the present disclosure, a model training device is provided, comprising:

[0019] A first acquisition module, used for inputting the sample facial image into the first image processing model, and acquiring the texture coefficient of the sample facial image output by the first image processing model;

[0020] A first generating module, for generating an initial texture map of a sample facial image based on a texture coefficient and a texture base;

[0021] A second acquisition module is used to input the target area map extracted from the sample facial image into the second image processing model to obtain the target object offset feature map of the sample facial image output by the second image processing model;

[0022] A second generation module is used to obtain a final texture map based on the initial texture map and the target object offset feature map;

[0023] A third generation module, for generating a rendering of the sample facial image based on the final texture image;

[0024] A construction module for constructing a loss function based on the rendered image and the label rendered image of the sample face image;

[0025] The training module is used to adjust the parameters of the first image processing model and the second image processing model respectively based on the loss function.

[0026] According to a fourth aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0027] A fourth acquisition module, used for inputting the facial image to be processed into the first image processing model, and acquiring the texture coefficient of the facial image to be processed output by the first image processing model;

[0028] A fifth generating module, used for generating an initial texture map of the face image to be processed based on the texture coefficient and the texture base;

[0029] A fifth acquisition module, used for inputting the target area map extracted from the face image to be processed into the second image processing model, and acquiring the target object offset feature map of the face image to be processed output by the second image processing model;

[0030] A sixth generation module, used to obtain a final texture map based on the initial texture map and the target object offset feature map;

[0031] Among them, the first image processing model and the second image processing model are obtained by using the training method of the image processing model provided according to the first aspect.

[0032] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0033] at least one processor; and

[0034] a memory communicatively connected to the at least one processor; wherein,

[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods provided by the first and second aspects above.

[0036] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the methods provided in the first and second aspects above.

[0037] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the methods provided in the first and second aspects above when executed by a processor.

[0038] The disclosed embodiments can improve the similarity between the reconstructed result and the original face.

[0039] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0041] Figure 1 is a schematic diagram of a distributed cluster processing scenario according to an embodiment of the present disclosure;

[0042] Figure 2 The following is a flow chart of a training method for an image processing model according to an embodiment of the present disclosure. Figure 1 ;

[0043] Figure 3 The following is a flow chart of a training method for an image processing model according to an embodiment of the present disclosure. Figure 2 ;

[0044] Figure 4 is a schematic diagram of a framework for model training according to an embodiment of the present disclosure;

[0045] Figure 5 The process diagram of the image processing method according to the embodiment of the present disclosure is as follows Figure 1 ;

[0046] Figure 6 The process diagram of the image processing method according to the embodiment of the present disclosure is as follows Figure 2 ;

[0047] Figure 7 A schematic diagram of a scenario for model training provided in an embodiment of the present disclosure;

[0048] Figure 8 A schematic diagram of a scene for image processing provided by an embodiment of the present disclosure;

[0049] Fig. 9 is a structural schematic diagram of a training device for an image processing model according to an embodiment of the present disclosure;

[0050] Fig.10 is a schematic structural diagram of an image processing device according to an embodiment of the present disclosure;

[0051] Fig.11 It is a block diagram of an electronic device used to implement the training method of the image processing model or the image processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0053] The terms "first", "second", and "third", etc. in the specification embodiments and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0054] Figure 1This is a schematic diagram of a distributed cluster processing scenario according to an embodiment of the present disclosure. The distributed cluster system is an example of a cluster system. It exemplarily describes how to use the distributed cluster system for model training. The present disclosure is not limited to model training on a single machine or multiple machines. Distributed processing can further improve the accuracy of model training. Figure 1 As shown, the distributed cluster system includes multiple nodes (such as server cluster 101, server 102, server cluster 103, server 104, server 105, and server 105 can also be connected to electronic devices, such as mobile phone 1051 and laptop computer 1052), and one or more model training tasks can be performed together between multiple nodes, and between multiple nodes and connected electronic devices. Optionally, multiple nodes in the distributed cluster system can adopt a data-parallel model training method, and multiple nodes can perform model training tasks based on the same training method to better train the model; if multiple nodes in the distributed cluster system adopt a model-parallel model training method, multiple nodes can perform model training tasks based on different training methods to better train the model. Optionally, after each round of model training is completed, data exchange (such as data synchronization) can be performed between multiple nodes.

[0055] The present disclosure provides a method for training an image processing model. Figure 2 It is a flowchart of a method for training an image processing model according to an embodiment of the present disclosure. The method can be applied to a training device for an image processing model. For example, the device can be deployed in a terminal or server or other processing device in a single machine, multiple machines or cluster system to implement model training and other processing. The terminal can be a user equipment (UE, User Equipment), a mobile device, a personal digital assistant (PDA, Personal Digital Assistant), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. In some possible implementations, the method can be applied to Figure 1 In any node or electronic device (mobile phone or desktop computer, etc.) in the cluster system shown. Figure 2 As shown, the training method of the image processing model includes:

[0056] S201, inputting a sample facial image into a first image processing model, and obtaining a texture coefficient of the sample facial image output by the first image processing model;

[0057] S202, generating an initial texture map of the sample face image based on the texture coefficient and the texture base;

[0058] S203, inputting the target area map extracted from the sample facial image into the second image processing model, and obtaining the target object offset feature map of the sample facial image output by the second image processing model;

[0059] S204, obtaining a final texture map based on the initial texture map and the target object offset feature map;

[0060] S205, generating a rendering of the sample facial image based on the final texture image;

[0061] S206, constructing a loss function based on the rendered image and the label rendered image of the sample facial image;

[0062] S207: Adjust parameters of the first image processing model and the second image processing model respectively based on the loss function.

[0063] Among them, the sample face image is the sample image used for model training.

[0064] Among them, the sample facial image can be a facial image of a target body, wherein the target body can be a person or an animal, and the sample facial image can be a facial image collected online, for example, the facial image of the target body can be collected online through web crawler technology, or the sample facial image can also be a facial image collected offline, or the sample facial image can also be a facial image of the target body collected in real time, or the sample facial image can also be an artificially synthesized facial image, etc., and the embodiments of the present disclosure do not impose any restrictions on this.

[0065] The target area map is a part of the sample face map. The target area map includes a map of the area where the target object is located. Here, the target object includes at least one of eyebrows, eyes, nose, mouth, ears, and lips.

[0066] Among them, the first image processing model includes at least one network structure, and the at least one network structure includes a convolutional neural network (CNN).

[0067] The second image processing model includes at least one network structure, and the at least one network structure includes CNN.

[0068] Among them, the number of network layers of the first image processing model is greater than the number of network layers of the second image processing model.

[0069] The texture base may be a texture base of a single reconstruction style or a texture base of multiple reconstruction styles. The reconstruction styles include but are not limited to real-life style, cartoon style, funny style, etc., which are not limited in the present disclosure.

[0070] The texture base may be an open source texture base or a private texture base obtained after multiple rounds of texture base extraction and updating. The present disclosure does not limit the source of the texture base.

[0071] In some embodiments, the target region map includes an eyebrow region map, and extracting the eyebrow region map from the sample facial image includes: obtaining two eyebrow key points of the sample facial image through an open source algorithm, deducting an external quadrilateral area, and obtaining the eyebrow region map. Here, the open source algorithm is a facial key point detection algorithm.

[0072] In some embodiments, the target area map includes an eyebrow area map, and extracting the eyebrow area map from the sample facial image includes: inputting the sample facial image into a trained detection model to obtain the eyebrow area map output by the detection model.

[0073] In some embodiments, in order to improve the accuracy of the first image processing model, the style of the texture base and the style of the label rendering image are the same style.

[0074] In facial three-dimensional reconstruction, eyebrows are generated by texture map reconstruction, so the accuracy of the extracted eyebrow shape will affect the accuracy of the reconstructed texture map. In the related art, the eyebrow boundary in the texture image is blurred, the generalization is poor, and the reconstructed eyebrow shape has a low similarity with the original eyebrow shape. In some embodiments, the target area map extracted from the sample facial image is input into the second image processing model to obtain the target object offset feature map of the sample facial image output by the second image processing model, including: inputting the eyebrow area map extracted from the sample facial image into the second image processing model to obtain the eyebrow shape offset feature map of the sample facial image output by the second image processing model. In this way, at least the eyebrows in the texture map can be made clearer, the transferability and generalization of texture maps of different reconstruction styles can be improved, and the recognition of the texture map can be improved, thereby improving the similarity between the reconstruction result, such as the rendering, and the original face.

[0075] In some embodiments, inputting the eyebrow region map extracted from the sample facial image into the second image processing model, and obtaining the eyebrow shape offset feature map of the sample facial image output by the second image processing model includes:

[0076] The eyebrow region map is input into the second image processing model, and the dimension is reduced to n*m through the full convolution layer, and then increased to a pre-set p*q to obtain the eyebrow shape offset feature map; where n, m, p and q are all integers. For example, n=2, m=4; p=64, q=128. For example, the texture base uses a fixed eyebrow position, and the size of the rectangular frame of the eyebrow region is 64*128. In the generated texture image, the region is compared with the predicted eyebrow shape offset feature map. Figure 1 Substitute it into the normalization function, and the eyebrow shape of this area on the texture image will automatically generate the eyebrow shape effect of the sample facial image, and it can be trained iteratively.

[0077] It can be understood that the target area map may include not only the eyebrow area map, but also other area maps such as the mouth area map, the nose area map, and the eye area map. The above target object offset feature map may include not only the eyebrow shape offset feature map, but also other shape offset feature maps such as the mouth shape offset feature map, the nose shape offset feature map, and the eye shape offset feature map, which are not listed in detail here.

[0078] The technical solution described in the embodiment of the present disclosure is as follows: when training the first image processing model and the second image processing model, the target area map extracted from the sample facial image is input into the second image processing model to obtain the target object offset feature map of the sample facial image output by the second image processing model; based on the initial texture map and the target object offset feature map, a final texture map is obtained; based on the final texture map, a rendering of the sample facial image is generated; based on the rendering map and the label rendering map of the sample facial image, a loss function is constructed; based on the loss function, the parameters of the first image processing model and the second image processing model are adjusted respectively, so that at least the target object in the texture map can be made clearer, the transferability and generalization of texture maps of different reconstruction styles can be improved, the recognizability of the texture map can be improved, and thus the similarity between the reconstruction results, such as the rendering map and the original face can be improved.

[0079] In some embodiments, constructing a loss function based on the rendered image and the label rendered image of the sample face image includes:

[0080] Determining a first feature map of the rendered image and a second feature map of the label rendered image;

[0081] A loss function is constructed based on the first feature map of the rendering image and the second feature map of the label rendering image.

[0082] In some embodiments, determining a first feature map of a rendering and a second feature map of a label rendering includes: inputting the rendering and the label rendering into a pre-trained facial recognition model to obtain a first feature map of the rendering and a second feature map of the label rendering output by the facial recognition model.

[0083] The first feature map is a map of facial features extracted from the sample face map, for example, the first feature map is an eye feature map, or the first feature map is a nose feature map, or the first feature map is a mouth feature map. Similarly, the second feature map is a map of facial features extracted from the label face map, for example, the second feature map is an eye feature map, or the second feature map is a nose feature map, or the second feature map is a mouth feature map.

[0084] It should be noted that the first feature map and the second feature map target the same facial area, that is, when the first feature map is an eye feature map, the second feature map is also an eye feature map; when the first feature map is a mouth feature map, the second feature map is also a mouth feature map.

[0085] In the disclosed embodiments, there is no limitation on how to train the facial recognition model. Any model that can recognize facial features in an image can be used as a facial recognition model.

[0086] It is understood that determining the first feature map of the rendering and the second feature map of the label rendering is not limited to inputting the rendering and the label rendering into the facial recognition model. For example, facial features of the rendering and the label rendering can be extracted by a feature extraction algorithm to obtain the first feature map of the rendering and the second feature map of the label rendering.

[0087] In this way, constructing a loss function based on the first feature map of the rendered image and the second feature map of the label rendered image helps to improve the recognition of the texture image output by the first image processing model, and also helps to improve the accuracy of the target object offset feature map output by the second image processing model, making the target object in the texture image clearer and more accurate, thereby improving the similarity between the reconstructed result and the original face.

[0088] The present disclosure provides a method for training an image processing model. Figure 3 This is a flowchart of a method for training an image processing model according to an embodiment of the present disclosure. The method can be applied to a training device for an image processing model. For example, the device can be deployed in a terminal or server or other processing device in a single machine, multiple machines or cluster system to implement model training and other processing. The terminal can be a user device, a mobile device, a personal digital processing, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. In some possible implementations, the method can be applied to Figure 1 In any node or electronic device (mobile phone or desktop computer, etc.) in the cluster system shown. Figure 3 As shown, the training method of the image processing model includes:

[0089] S301, inputting a sample facial image into a first image processing model, and obtaining a texture coefficient of the sample facial image output by the first image processing model;

[0090] S302, generating an initial texture map of the sample face image based on the texture coefficient and the texture base;

[0091] S303, obtaining a shape coefficient of the sample facial image output by the first image processing model;

[0092] S304, generating a shape model of the sample facial image based on the shape coefficient and the shape basis;

[0093] S305, inputting the target area map extracted from the sample facial image into the second image processing model, and obtaining the target object offset feature map of the sample facial image output by the second image processing model;

[0094] S306, obtaining a final texture map based on the initial texture map and the target object offset feature map;

[0095] S307, generating a rendering of the sample facial image based on the final texture image and the shape model;

[0096] S308, constructing a loss function based on the rendered image and the label rendered image of the sample facial image;

[0097] S309: Adjust the parameters of the first image processing model and the second image processing model respectively based on the loss function.

[0098] Among them, S304 is executed after S303, but S303 can be executed after S302, can also be executed before S302, and can also be executed at the same time as S302.

[0099] In the embodiment of the present disclosure, the dimension of the shape base is not limited. It can be understood that the higher the dimension of the shape base, the higher the precision of the shape model.

[0100] The shape base may be a shape base of a single reconstruction style or a shape base of multiple reconstruction styles. The reconstruction styles include but are not limited to real-life style, cartoon style, funny style, etc., and the present disclosure does not limit this.

[0101] The shape base may be an open source shape base or a private shape base obtained after multiple rounds of shape base extraction and updating. The embodiment of the present disclosure does not limit the source of the shape base.

[0102] In some embodiments, in order to improve the accuracy of the first image processing model and the second image processing model, the style of the shape base and the style of the texture base are the same.

[0103] In other embodiments, the same shape substrate corresponds to texture substrates of various styles.

[0104] In this way, when training the first image processing model and the second image processing model, the texture coefficient and texture basis, the shape coefficient and shape basis, and the target object offset feature are considered, which weakens the problem of insufficient or incomplete sources of texture basis and shape basis, and improves the transferability and generalization of texture basis and shape basis. It can not only improve the recognition of the texture image output by the first image processing model and the recognition of the shape model, but also improve the accuracy of the target object offset feature output by the second image processing model, thereby further improving the recognition of the rendered image and improving the similarity between the reconstructed result and the original face.

[0105] In some embodiments, generating a rendering of the sample facial image based on the final texture image and the shape model includes:

[0106] The final texture map and the shape model are input into a renderer to obtain a rendering of the sample facial image output by the renderer.

[0107] Here, the renderer may be a differentiable renderer.

[0108] It is understood that determining the rendering is not limited to the implementation of inputting the final texture map and shape model into the renderer. For example, the final texture map and shape model may be rendered by a rendering algorithm to obtain the rendering.

[0109] In this way, the rendering image can be quickly determined, which helps save time in obtaining the rendering image, thereby helping to increase the speed of training the first image processing model and the second image processing model.

[0110] In some embodiments, adjusting parameters of the first image processing model and the second image processing model based on the loss function respectively includes:

[0111] Determine loss values ​​of the first feature map and the second feature map corresponding to the sample facial image based on the loss function;

[0112] Based on the loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the loss value drops to a first preset range.

[0113] Here, the first image processing model includes a multi-layer network. When training the first image processing model, it is necessary to adjust the parameters of at least one or more layers in the multi-layer network.

[0114] Here, the first preset range is [0, x], and x is a positive number greater than 0. The value of x can be set or adjusted according to accuracy requirements or speed requirements.

[0115] In this way, based on the loss value of a single sample facial image, the parameters of each layer of the network of the first image processing model and the second image processing model are adjusted, which can save the training time of the first image processing model and the second image processing model, improve the training speed of the first image processing model and the second image processing model, and enable the trained first image processing model and the second image processing model to better adapt to the reconstruction style of the same type as the sample facial image.

[0116] In some embodiments, adjusting parameters of the first image processing model and the second image processing model based on the loss function respectively includes:

[0117] Determine loss values ​​of first feature maps and second feature maps respectively corresponding to the plurality of sample facial images based on the loss function;

[0118] Determine a total loss value of the plurality of sample facial images based on the loss values ​​respectively corresponding to the plurality of sample facial images;

[0119] Based on the total loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the total loss value drops to a second preset range.

[0120] Here, the first image processing model and the second image processing model include a multi-layer network, and the number of network layers of the second image processing model is less than the number of network layers of the first image processing model. When training the first image processing model and the second image processing model, it is necessary to adjust the parameters of at least one or more layers in the first image processing model and the second image processing model.

[0121] Here, the second preset range is [0, y], and y is a positive number greater than 0. The value of y can be set or adjusted according to accuracy requirements or speed requirements.

[0122] In this way, based on the total loss value of multiple sample facial images, the parameters of each layer of the network of the first image processing model and the second image processing model are adjusted, which can improve the generalization and transferability of the first image processing model and the second image processing model, and enable the trained first image processing model and the second image processing model to adapt to various reconstruction styles.

[0123] Figure 4 A schematic diagram of the model training architecture is shown, Figure 4As shown, the facial image is input into the first image processing model to obtain the texture coefficient; the texture map is generated according to the texture base and the texture coefficient; the target area map extracted from the sample facial image is input into the second image processing model to obtain the target object offset feature map of the sample facial image output by the second image processing model; based on the initial texture map and the target object offset feature map, a normalization function such as the grid_sample function is substituted to obtain the final texture map; the final texture map is input into the renderer to obtain the rendered image; the loss function is constructed based on the rendered image and the label rendered image of the sample facial image; the parameters of the first image processing model and the second image processing model are adjusted according to the loss function.

[0124] It should be understood that Figure 4 The architecture diagram shown is only schematic, and those skilled in the art can Figure 4 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0125] Based on the first image processing model and the second image processing model obtained by training the above-mentioned image processing model training method, the embodiment of the present disclosure provides an image processing method, which is applied to electronic devices, including but not limited to computers, mobile phones or tablet computers, etc. The present disclosure does not limit the type of electronic devices. Figure 5 As shown, the image processing method includes:

[0126] S501, inputting a facial image to be processed into a first image processing model, and obtaining a texture coefficient of the facial image to be processed output by the first image processing model;

[0127] S502, generating an initial texture map of the face image to be processed based on the texture coefficient and the texture base;

[0128] S503, inputting the target area map extracted from the face image to be processed into the second image processing model, and obtaining the target object offset feature map of the face image to be processed output by the second image processing model;

[0129] S504, obtaining a final texture map based on the initial texture map and the target object offset feature map.

[0130] Here, the facial image to be processed is a facial image that needs to be processed.

[0131] In some implementations, generating an initial texture map of the facial image to be processed based on texture coefficients and texture bases includes: generating the initial texture map based on a linear sum of the texture coefficients and the texture bases.

[0132] The technical solution described in the embodiment of the present disclosure obtains a final texture map based on the initial texture map and the target object offset feature map, which can make the target object in the texture image clearer and more accurate, and keep the shape consistent with the target object of the original face, thereby improving the similarity between the reconstruction result, such as the rendering, and the original face.

[0133] Based on the first image processing model and the second image processing model obtained by training the above-mentioned image processing model training method, the embodiment of the present disclosure provides an image processing method, which is applied to electronic devices, including but not limited to computers, mobile phones or tablet computers, etc. The present disclosure does not limit the type of electronic devices. Figure 6 As shown, the image processing method includes:

[0134] S601, inputting a facial image to be processed into a first image processing model, and obtaining a texture coefficient of the facial image to be processed output by the first image processing model;

[0135] S602, generating an initial texture map of the face image to be processed based on the texture coefficient and the texture base;

[0136] S603, obtaining a shape coefficient of the facial image to be processed output by the first image processing model;

[0137] S604, generating a shape model of the face image to be processed based on the shape coefficient and the shape basis;

[0138] S605, inputting the target area map extracted from the face image to be processed into the second image processing model, and obtaining the target object offset feature map of the face image to be processed output by the second image processing model;

[0139] S606, obtaining a final texture map based on the initial texture map and the target object offset feature map;

[0140] S607, obtaining a rendering of the facial image to be processed based on the final texture image and the shape model.

[0141] Among them, S604 is executed after S603, but S603 can be executed after S602, can also be executed before S602, and can also be executed at the same time as S602.

[0142] In some embodiments, the target region map extracted from the facial image to be processed is input into the second image processing model, and the target object offset feature map of the facial image to be processed output by the second image processing model is obtained, including: the eyebrow region map extracted from the facial image to be processed is input into the second image processing model, and the eyebrow shape offset feature map of the facial image to be processed output by the second image processing model is obtained. In this way, the eyebrows in the texture image can be made clearer and more accurate, and consistent with the eyebrow shape of the original face, thereby improving the similarity between the reconstruction result, such as the rendering image, and the original face.

[0143] The technical solution described in the embodiments of the present disclosure can make the target object in the texture image clearer and more accurate, and consistent with the shape of the original facial target object. It can not only improve the recognition of the texture image of the facial image to be processed, but also improve the recognition of the shape model of the facial image to be processed, thereby further improving the recognition of the rendered image and improving the similarity of the reconstruction result, such as the rendered image and the original face.

[0144] The present disclosure also provides a schematic diagram of a model training scenario, such as Figure 7 As shown, an electronic device such as a cloud server mines sample facial images and labeled facial images that meet the training tasks from multiple data sources according to the training tasks sent by the terminal. Here, the training tasks sent by different terminals can be training tasks for different reconstruction styles. The required sample facial images and labeled facial images may be different for training tasks with different reconstruction styles. The electronic device inputs the sample facial image into the first image processing model to obtain the texture coefficient of the sample facial image; generates the initial texture map of the sample facial image based on the texture coefficient and the texture base; inputs the eyebrow area map extracted from the sample facial image into the second image processing model to obtain the eyebrow shape offset feature map of the sample facial image; obtains the final texture map based on the initial texture map and the eyebrow shape offset feature map; generates a rendering of the sample facial image based on the final texture map; constructs a loss function based on the rendering map and the label rendering map of the sample facial image; adjusts the parameters of the first image processing model and the second image processing model based on the loss function; and the terminal returns the trained first image processing model and the second image processing model of the corresponding reconstruction style.

[0145] The present disclosure also provides a schematic diagram of an image processing scenario. Figure 8 As shown, an electronic device such as a cloud server receives a facial image to be processed sent by a terminal, determines a texture map and / or a rendering image based on the facial image to be processed and the reconstruction style sent by the terminal, and returns a texture map and / or a rendering image corresponding to the reconstruction style to the terminal.

[0146] The present disclosure does not limit the number of terminals and electronic devices, and actual applications may include multiple terminals and multiple electronic devices.

[0147] It should be understood that Figure 7 and Figure 8 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 7 and Figure 8 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0148] It should be noted that the training method and image processing method of the image processing model disclosed in the present invention are not a head model for a specific user and cannot reflect the personal information of a specific user.

[0149] The present disclosure provides a training device for an image processing model. Fig. 9 As shown, the training device of the image processing model may include:

[0150] A first acquisition module 910, configured to input the sample facial image into a first image processing model, and acquire a texture coefficient of the sample facial image output by the first image processing model;

[0151] A first generating module 920, for generating an initial texture map of the sample face map based on the texture coefficient and the texture base;

[0152] A second acquisition module 930 is used to input the target area map extracted from the sample facial image into the second image processing model to obtain the target object offset feature map of the sample facial image output by the second image processing model;

[0153] A second generating module 940 is used to obtain a final texture map based on the initial texture map and the target object offset feature map;

[0154] A third generating module 950 is used to generate a rendering of the sample facial image based on the final texture image;

[0155] A construction module 960 is used to construct a loss function based on the rendered image and the label rendered image of the sample face image;

[0156] The training module 970 is used to adjust the parameters of the first image processing model and the second image processing model respectively based on the loss function.

[0157] In some embodiments, the second acquisition module 930 is specifically used to:

[0158] The eyebrow region map extracted from the sample facial image is input into the second image processing model, and the eyebrow shape offset feature map of the sample facial image output by the second image processing model is obtained.

[0159] In some embodiments, the building block 960 is specifically configured to:

[0160] Determining a first feature map of the rendered image and a second feature map of the label rendered image;

[0161] A loss function is constructed based on the first feature map of the rendering image and the second feature map of the label rendering image.

[0162] In some embodiments, the training device of the image processing model may further include:

[0163] A third acquisition module, used to acquire the shape coefficient of the sample facial image output by the first image processing model;

[0164] The fourth generating module is used to generate a shape model of the sample face image based on the shape coefficient and the shape basis.

[0165] In some embodiments, the fourth generating module is specifically used to:

[0166] The final texture map and the shape model are input into a renderer to obtain a rendering of the sample facial image output by the renderer.

[0167] In some embodiments, the training module 970 is specifically used to:

[0168] Determine loss values ​​of the first feature map and the second feature map corresponding to the sample facial image based on the loss function;

[0169] Based on the loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the loss value drops to a first preset range.

[0170] In some embodiments, the training module 970 is specifically used to:

[0171] Determine loss values ​​of first feature maps and second feature maps respectively corresponding to the plurality of sample facial images based on the loss function;

[0172] Determine a total loss value of the plurality of sample facial images based on the loss values ​​respectively corresponding to the plurality of sample facial images;

[0173] Based on the total loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the total loss value drops to a second preset range.

[0174] Those skilled in the art should understand that the functions of each processing module in the training device of the image processing model of the embodiment of the present disclosure can be understood with reference to the relevant description of the training method of the image processing model mentioned above, and each processing module in the training device of the image processing model of the embodiment of the present disclosure can be realized by an analog circuit that realizes the functions of the embodiment of the present disclosure, and can also be realized by running software that executes the functions of the embodiment of the present disclosure on an electronic device.

[0175] The training device of the image processing model of the embodiment of the present disclosure can make the target object in the texture image clearer, improve the transferability and generalization of the target object in texture images of different reconstruction styles, improve the recognition of the texture image, and thus improve the similarity between the reconstruction result and the original face.

[0176] The present disclosure provides an image processing device, such as Fig.10As shown, the image processing device may include:

[0177] A fourth acquisition module 1010 is used to input the facial image to be processed into the first image processing model, and obtain the texture coefficient of the facial image to be processed output by the first image processing model;

[0178] A fifth generating module 1020, configured to generate an initial texture map of the facial image to be processed based on the texture coefficient and the texture base;

[0179] A fifth acquisition module 1030 is used to input the target area map extracted from the face image to be processed into the second image processing model, and obtain the target object offset feature map of the face image to be processed output by the second image processing model;

[0180] A sixth generating module 1040, configured to obtain a final texture map based on the initial texture map and the target object offset feature map;

[0181] Among them, the first image processing model and the second image processing model are obtained according to the training method of the image processing model above.

[0182] In some embodiments, the fifth acquisition module 1030 is specifically configured to:

[0183] The eyebrow region map extracted from the facial image to be processed is input into the second image processing model, and the eyebrow shape offset feature map of the facial image to be processed output by the second image processing model is obtained.

[0184] In some embodiments, the image processing device may also include: a sixth acquisition module, used to obtain the shape coefficient of the facial image to be processed output by the first image processing model; a seventh generation module, used to generate a shape model of the facial image to be processed based on the shape coefficient and the shape basis; an eighth generation module, used to obtain a rendering of the facial image to be processed based on the final texture map and the shape model.

[0185] Those skilled in the art should understand that the functions of each processing module in the image processing device of the embodiment of the present disclosure can be understood with reference to the relevant description of the aforementioned image processing method, and each processing module in the image processing device of the embodiment of the present disclosure can be realized by an analog circuit that realizes the functions described in the embodiment of the present disclosure, or can be realized by running software that executes the functions described in the embodiment of the present disclosure on an electronic device.

[0186] The image processing device of the disclosed embodiment can make the target object in the texture image clearer and more accurate, and consistent with the shape of the original facial target object. It can not only improve the recognition of the texture image of the facial image to be processed, but also improve the recognition of the shape model of the facial image to be processed, thereby further improving the recognition of the rendered image and improving the similarity between the reconstructed result and the original face.

[0187] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0188] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0189] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0190] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0191] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0192] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the training method or image processing method of the image processing model. For example, in some embodiments, the training method or image processing method of the image processing model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the training method of the image processing model or the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the training method of the image processing model or the image processing method in any other appropriate manner (e.g., by means of firmware).

[0193] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0195] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0197] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0198] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on the respective computers and having a client and server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0199] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0200] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for an image processing model, comprising: Inputting the sample facial image into a first image processing model, and obtaining a texture coefficient of the sample facial image output by the first image processing model; generating an initial texture map of the sample face image based on the texture coefficients and the texture basis; Inputting the target area map extracted from the sample facial image into a second image processing model, and obtaining a target object offset feature map of the sample facial image output by the second image processing model; Obtaining a final texture map based on the initial texture map and the target object offset feature map; generating a rendering of the sample facial image based on the final texture image; Constructing a loss function based on the rendered image and the labeled rendered image of the sample facial image; The parameters of the first image processing model and the second image processing model are adjusted respectively based on the loss function.

2. The method according to claim 1, wherein: The step of inputting the target area map extracted from the sample facial image into a second image processing model, and obtaining the target object offset feature map of the sample facial image output by the second image processing model, comprises: The eyebrow region map extracted from the sample facial image is input into a second image processing model, and an eyebrow shape offset feature map of the sample facial image output by the second image processing model is obtained.

3. The method according to claim 1, wherein: The constructing a loss function based on the rendered image and the label rendered image of the sample facial image includes: Determining a first characteristic map of the rendering and a second characteristic map of the label rendering; Constructing a loss function based on the first feature map of the rendering image and the second feature map of the label rendering image; The first feature map is a map of facial features extracted from the sample facial map, and the second feature map is a map of facial features extracted from a label facial map corresponding to the sample facial map.

4. The method according to claim 3, wherein: The adjusting the parameters of the first image processing model and the second image processing model respectively based on the loss function includes: Determine loss values ​​of the first feature map and the second feature map corresponding to the sample facial image based on the loss function; Based on the loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the loss value drops to a first preset range.

5. The method according to claim 3, wherein: The adjusting the parameters of the first image processing model and the second image processing model respectively based on the loss function includes: Determine, based on the loss function, loss values ​​of the first feature maps and the second feature maps respectively corresponding to the plurality of sample facial images; Determine a total loss value of the plurality of sample facial images based on the loss values ​​respectively corresponding to the plurality of sample facial images; Based on the total loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the total loss value drops to a second preset range.

6. The method according to claim 1, further comprising: Obtaining a shape coefficient of the sample facial image output by the first image processing model; A shape model of the sample face image is generated based on the shape coefficients and the shape basis.

7. The method according to claim 6, wherein: The step of generating a rendering of the sample facial image based on the final texture image comprises: The final texture map and the shape model are input into a renderer to obtain a rendering of the sample facial image output by the renderer.

8. An image processing method, comprising: Inputting a facial image to be processed into a first image processing model, and obtaining a texture coefficient of the facial image to be processed output by the first image processing model; Generate an initial texture map of the face image to be processed based on the texture coefficient and the texture base; Inputting the target area map extracted from the face image to be processed into a second image processing model, and obtaining a target object offset feature map of the face image to be processed output by the second image processing model; Obtaining a final texture map based on the initial texture map and the target object offset feature map; Wherein, the first image processing model and the second image processing model are obtained by using the training method of the image processing model according to any one of claims 1 to 7.

9. The method according to claim 8, wherein: The step of inputting the target area map extracted from the face image to be processed into a second image processing model, and obtaining the target object offset feature map of the face image to be processed output by the second image processing model, comprises: The eyebrow region map extracted from the facial image to be processed is input into a second image processing model, and an eyebrow shape offset feature map of the facial image to be processed output by the second image processing model is obtained.

10. The method according to claim 8, further comprising: Obtaining a shape coefficient of the facial image to be processed output by the first image processing model; generating a shape model of the face image to be processed based on the shape coefficient and the shape basis; A rendering of the facial image to be processed is obtained based on the final texture image and the shape model.

11. A model training device for an image processing model, comprising: A first acquisition module, used for inputting a sample facial image into a first image processing model, and acquiring a texture coefficient of the sample facial image output by the first image processing model; A first generating module, configured to generate an initial texture map of the sample facial image based on the texture coefficient and the texture base; A second acquisition module, used for inputting a target area map extracted from the sample facial image into a second image processing model, and acquiring a target object offset feature map of the sample facial image output by the second image processing model; A second generating module, configured to obtain a final texture map based on the initial texture map and the target object offset feature map; A third generating module, configured to generate a rendering of the sample facial image based on the final texture image; A construction module, configured to construct a loss function based on the rendering image and the label rendering image of the sample facial image; A training module is used to adjust the parameters of the first image processing model and the second image processing model respectively based on the loss function.

12. The device according to claim 11, wherein: The second acquisition module is used to: The eyebrow region map extracted from the sample facial image is input into a second image processing model, and an eyebrow shape offset feature map of the sample facial image output by the second image processing model is obtained.

13. The device according to claim 11, wherein: The building blocks are used to: Determining a first characteristic map of the rendering and a second characteristic map of the label rendering; Constructing a loss function based on the first feature map of the rendering image and the second feature map of the label rendering image; The first feature map is a map of facial features extracted from the sample facial map, and the second feature map is a map of facial features extracted from a label facial map corresponding to the sample facial map.

14. The device according to claim 13, wherein: The training module is used to: Determine loss values ​​of the first feature map and the second feature map corresponding to the sample facial image based on the loss function; Based on the loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the loss value drops to a first preset range.

15. The device according to claim 13, wherein: The training module is used to: Determine, based on the loss function, loss values ​​of the first feature maps and the second feature maps respectively corresponding to the plurality of sample facial images; Determine a total loss value of the plurality of sample facial images based on the loss values ​​respectively corresponding to the plurality of sample facial images; Based on the total loss value, the parameters of each layer network of the first image processing model and the parameters of each layer network of the second image processing model are adjusted until the total loss value drops to a second preset range.

16. The apparatus according to claim 11, further comprising: A third acquisition module, used to acquire the shape coefficient of the sample facial image output by the first image processing model; A fourth generating module is used to generate a shape model of the sample facial image based on the shape coefficient and the shape basis.

17. The device according to claim 16, wherein: The fourth generating module is used for: The final texture map and the shape model are input into a renderer to obtain a rendering of the sample facial image output by the renderer.

18. An image processing device, comprising: A fourth acquisition module, used for inputting the facial image to be processed into the first image processing model, and acquiring the texture coefficient of the facial image to be processed output by the first image processing model; A fifth generating module, configured to generate an initial texture map of the face image to be processed based on the texture coefficient and the texture base; A fifth acquisition module, configured to input the target area map extracted from the face image to be processed into a second image processing model, and acquire a target object offset feature map of the face image to be processed output by the second image processing model; A sixth generating module, configured to obtain a final texture map based on the initial texture map and the target object offset feature map; Wherein, the first image processing model and the second image processing model are obtained by using the training method of the image processing model according to any one of claims 1 to 7.

19. The device according to claim 18, wherein The fifth acquisition module is used for: The eyebrow region map extracted from the facial image to be processed is input into a second image processing model, and an eyebrow shape offset feature map of the facial image to be processed output by the second image processing model is obtained.

20. The apparatus of claim 18, further comprising: A sixth acquisition module, used for acquiring a shape coefficient of the facial image to be processed output by the first image processing model; A seventh generating module, configured to generate a shape model of the face image to be processed based on the shape coefficient and the shape basis; An eighth generating module is used to obtain a rendering image of the facial image to be processed based on the final texture image and the shape model.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images