Method, apparatus, and system for processing facial images

Abstract

The present application discloses a method, apparatus, and system for processing facial images. Among them, the method includes: obtaining a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; and adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object. The present application solves the technical problem in the related art that generating virtual objects manually results in high processing costs and poor effects.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a method, apparatus, and system for processing facial images. Background Art

[0002] With the increasing status of live streaming and short videos in online media and the popularization of three-dimensional rendering technology, the demand for obtaining three-dimensional virtual avatars of physical objects has become increasingly significant. In traditional three-dimensional virtual avatar face sculpting systems, manual processing methods are usually adopted, where designers directly adjust the virtual avatar based on facial images and experience to make the virtual avatar more conform to the human face. However, the manual processing method has great disadvantages in terms of costs such as time and expenses, and it is also very difficult to ensure flexibility in use, and subjective cognitive differences among different designers may also result in different virtual avatars being produced from the same facial image.

[0003] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present application provide a method, apparatus, and system for processing facial images to at least solve the technical problem in the related art that generating virtual objects manually results in relatively high processing costs and poor effects.

[0005] According to one aspect of the embodiments of the present application, a method for processing a facial image is provided, including: obtaining a facial image of a physical object, where the facial image includes images of multiple facial components of the physical object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; and adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0006] According to another aspect of the embodiments of the present application, a method for processing a facial image is further provided, including: receiving a facial image of a physical object, where the facial image includes images of multiple facial components of the physical object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; and outputting the target virtual object.

[0007] According to another aspect of the embodiments of the present application, there is also provided a method for processing a face image, including: displaying a face image of an entity object on an interaction interface, where the face image includes images of multiple face components of the entity object; if an image operation instruction is detected in any area of the interaction interface, triggering the generation of an initial virtual object based on the multiple face components in the face image, and constructing a three-dimensional model of the face image; displaying a target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

[0008] According to another aspect of the embodiments of the present application, there is also provided a method for processing a face image, including: obtaining a face image of an entity object by calling a first interface, where the first interface includes: a first parameter, the parameter value of the first parameter is the face image, and the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; outputting the target virtual object by calling a second interface, where the second interface includes: a second parameter, the parameter value of the second parameter is the target virtual object.

[0009] According to another aspect of the embodiments of the present application, there is also provided a device for processing a face image, including: an obtaining module, configured to obtain a face image of an entity object, where the face image includes images of multiple face components of the entity object; a first generating module, configured to generate an initial virtual object based on the multiple face components in the face image; a constructing module, configured to construct a three-dimensional model of the face image; a second generating module, configured to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0010] According to another aspect of the embodiments of the present application, there is also provided a device for processing a face image, including: a receiving module, configured to receive a face image of an entity object, where the face image includes images of multiple face components of the entity object; a first generating module, configured to generate an initial virtual object based on the multiple face components in the face image; a constructing module, configured to construct a three-dimensional model of the face image; a second generating module, configured to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; an output module, configured to output the target virtual object.

[0011] According to another aspect of the embodiments of the present application, there is also provided a processing device for facial images, including: a first display module, configured to display a facial image of an entity object on an interaction interface, where the facial image includes images of multiple facial components of the entity object; a trigger module, configured to trigger, if an image operation instruction is detected in any area of the interaction interface, to generate an initial virtual object based on the multiple facial components in the facial image, and construct a three-dimensional model of the facial image; a second display module, configured to display a target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

[0012] According to another aspect of the embodiments of the present application, there is also provided a processing device for facial images, including: a first calling module, configured to obtain a facial image of an entity object by calling a first interface, where the first interface includes: a first parameter, and the parameter value of the first parameter is the facial image, and the facial image includes images of multiple facial components of the entity object; a first generating module, configured to generate an initial virtual object based on the multiple facial components in the facial image; a constructing module, configured to construct a three-dimensional model of the facial image; a second generating module, configured to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; a second calling module, configured to output the target virtual object by calling a second interface, where the second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

[0013] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned processing method for facial images.

[0014] According to another aspect of the embodiments of the present application, there is also provided a processing terminal, including: a memory and a processor, where the processor is configured to run the program stored in the memory, and when the program runs, it executes the above-mentioned processing method for facial images.

[0015] According to another aspect of the embodiments of the present application, there is also provided a processing system for facial images, including: a processor; and a memory, connected to the processor, and configured to provide instructions for the processor to perform the following processing steps: obtain a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; generate an initial virtual object based on the multiple facial components in the facial image; construct a three-dimensional model of the facial image; adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0016] According to another aspect of the embodiments of the present application, there is also provided a method for processing a face image, including: when a communication request is received, obtaining a face image of an entity object, where the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; and presenting the target virtual object in a communication interface.

[0017] In the embodiments of the present application, after obtaining the face image of the entity object, an initial virtual object can be generated based on the multiple face components in the face image, and after constructing the three-dimensional model of the face image, the initial virtual object is adjusted based on the three-dimensional model to generate a target virtual object, achieving the purpose of virtual object generation. It is easy to notice that the target virtual object is a fine-tuning of the initial virtual object based on the constructed three-dimensional model, which can be automatically completed without the need for manual operation by professional designers, avoiding completely different processing results for the same face image due to different designers, achieving the technical effects of saving processing time and labor costs, improving the processing effect, and enhancing the user experience and favorability, thereby solving the technical problem in the related art that generating virtual objects manually results in relatively high processing costs and poor effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0019] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing a face image according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a first method for processing a face image according to an embodiment of the present application;

[0021] Figure 3 is a flowchart of an alternative method for processing a face image according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of a second method for processing a face image according to an embodiment of the present application;

[0023] Figure 5 is a flowchart of a third method for processing a face image according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of an alternative interaction interface according to an embodiment of the present application;

[0025] Figure 7 It is a flowchart of a fourth method for processing facial images according to an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of a first device for processing facial images according to an embodiment of the present application;

[0027] Figure 9 It is a schematic diagram of a second device for processing facial images according to an embodiment of the present application;

[0028] Figure 10 It is a schematic diagram of a third device for processing facial images according to an embodiment of the present application;

[0029] Figure 11 It is a schematic diagram of a fourth device for processing facial images according to an embodiment of the present application;

[0030] Figure 12 It is a flowchart of a fifth method for processing facial images according to an embodiment of the present application;

[0031] Figure 13 It is a structural block diagram of a computer terminal according to an embodiment of the present application. Detailed implementation manners

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] First, some nouns or terms that appear in the process of describing the embodiments of the present application are applicable to the following explanations:

[0035] Photo-based face sculpting: In a given 3D virtual avatar generation system, generate a 3D virtual avatar that is closest to the person in the image based on the person's image.

[0036] 3D virtual avatar: A 3D mesh model with textures, used to approximate a specific person.

[0037] Multi-task image classification network: A deep learning network that takes an image as input and outputs multiple classification labels.

[0038] 3dmm: 3D Morphable Model, a 3D deformable model that expresses the entire human body or part of the human body (such as the face) through a set of continuously adjustable parameters.

[0039] Blendshape: A technique for deforming a single network model to achieve combinations between multiple predefined shapes and any quantity. The degree of deformation is quantitatively reflected by the Blendshape value. In a 3D virtual avatar generation system, the face of the generated virtual avatar is expressed as a combination of Blendshape values of facial features.

[0040] Currently, there is a method of directly estimating the parameters of facial features based on the key points of the face and then corresponding them to Blendshape values. This method is easily affected by various interference factors such as lighting, face pose, and occluders in actual photos. In addition, since the actual 3D virtual avatar face sculpting system uses cartoon-style avatars, there are many problems in directly corresponding the facial feature parameters to Blendshape values. Randomly corresponding the Blendshape values of facial feature parameters often leads to a large difference between the face sculpting result and the real face in the photo, and even makes the face sculpting result not look like a normal face.

[0041] To solve the above problems, this application provides the following implementation solutions, so that a 3D virtual avatar as similar as possible to the face in the photo can be generated in an existing 3D virtual avatar face sculpting system based on the photo.

[0042] Embodiment 1

[0043] According to an embodiment of the present application, a method for processing a facial image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0044] The method embodiment provided by the embodiment of the present application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Figure 1The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing face images is shown. As Figure 1 shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ……, 102n in the figure) (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0045] It should be noted that the above one or more processors 102 and / or other data processing circuits are generally referred to as "data processing circuits" herein. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for processing face images in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned method for processing face images. The memory 104 may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor 102, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0048] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0049] It should be noted here that in some alternative embodiments, the above-mentioned Figure 1 shown computer device (or mobile device) may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 1 is only an example of a specific specific instance and is intended to illustrate the types of components that may exist in the above-mentioned computer device (or mobile device).

[0050] Under the above operating environment, the present application provides a method for processing a face image as Figure 2 shown. Figure 2 is a flowchart of a first method for processing a face image according to an embodiment of the present application. As Figure 2 shown, the method may include the following steps:

[0051] Step S202, obtaining a face image of an entity object, where the face image includes images of multiple face components of the entity object.

[0052] The entity object in the above steps may be a real object or creature that needs to be virtualized in terms of image. Since the present application is directed to the face virtualization of the entity object, in the embodiments of the present application, the above-mentioned entity object may be a real person, a robot, etc., but is not limited thereto. The face components in the above steps may refer to the five facial features of a human face, including face shape, eyebrows, eyes, nose, mouth, etc., and can be set according to the generation requirements of the virtual image.

[0053] In an alternative embodiment, the face part of the physical object can be directly photographed by a photographing device such as a camera, a camera phone, a mobile phone, a tablet computer, a laptop computer, etc., so as to obtain a face image of the physical object. In another alternative embodiment, the physical object can be photographed by the above-mentioned photographing device, and the face image can be obtained by processing the image such as cropping.

[0054] It should be noted that the virtual image generation device can be a mobile terminal such as the user's smart phone or tablet computer, or a computer terminal such as a laptop computer or a PC. The user can obtain the face image and process it by directly photographing the face image or selecting a pre-photographed face image. In order to reduce the computing amount of the mobile terminal or the computer terminal, the virtual image generation device can also be a server, and the user can upload the face image to the server for processing through the Internet, WIFI, 3G, 4G or 5G, etc.

[0055] Step S204, generating an initial virtual object based on multiple face components in the face image.

[0056] In an alternative embodiment, the face image can be processed by using image target detection technology, and the position of each face component can be marked in the face image, that is, the face image is processed by using a pre-trained neural network model, each face component is framed in the face image, and the type of each face component is recognized.

[0057] Moreover, different classifications can be defined for each part of the human face, including face shape, eyebrows, eyes, nose, mouth, etc. (for example, different eye types can be defined as phoenix eyes, standard eyes, round eyes, etc.), and three-dimensional face materials corresponding to each classification, that is, Blendshape values of the corresponding parts, are pre-produced.

[0058] After the type of each face component is recognized, the corresponding Blendshape value can be read from the material library and aggregated into an initial virtual object.

[0059] Step S206, constructing a three-dimensional model of the face image.

[0060] In an alternative embodiment, the three-dimensional model of the face image can be directly constructed by using modeling means such as structured light. However, since this method requires the use of expensive depth sensors such as infrared and visible structured light, the processing cost is increased.

[0061] Therefore, in order to further reduce the cost, in the embodiment of the present application, a three-dimensional deformable model can be used to construct the three-dimensional model of the face image. In an alternative embodiment, the single photo face reconstruction technology based on 3dmm can be used to perform three-dimensional reconstruction on the face image to obtain a three-dimensional model.

[0062] Step S208: Adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0063] In an optional embodiment, on the basis of the initial virtual object, a three-dimensional model based on the facial image is fine-tuned so that the final target virtual object is more consistent with the facial image and has stronger uniqueness and difference.

[0064] Through the solution provided by the above-mentioned embodiment of the present application, after obtaining a facial image of a physical object, an initial virtual object can be generated based on multiple facial components in the facial image. After constructing a three-dimensional model of the facial image, the initial virtual object is adjusted based on the three-dimensional model to generate a target virtual object, thereby achieving the purpose of virtual object generation. It is easy to notice that the target virtual object is a fine-tuning of the initial virtual object based on the constructed three-dimensional model. This can be completed automatically without the need for manual operation by professional designers. This avoids the situation where different designers produce completely different processing results for the same facial image, achieving the technical effect of saving processing time and labor costs, improving processing effects, and enhancing user experience and favorability. This solves the technical problem of manually generating virtual objects in related technologies, which leads to high processing costs and poor results.

[0065] In the above embodiment of the present application, adjusting the initial virtual object based on the three-dimensional model to generate the target virtual object includes: adjusting the mixed shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted mixed shape parameters; and aggregating the adjusted mixed shape parameters to generate the target virtual object.

[0066] The blend shape parameters in the above steps may be Blendshape values. The virtual object may be expressed as a combination of the Blendshape values of the facial features.

[0067] In an optional embodiment, the initial virtual object is generated based on multiple facial components, rather than generating differentiated images for different characters. To ensure that the final target virtual object more closely matches the facial image, a 3D facial model can be constructed that accurately represents the character's facial features. Furthermore, the initial virtual object can be represented as a combination of BlendShape values. Therefore, the BlendShape values of the initial virtual object can be adjusted using the 3D model, and the adjusted BlendShape values can be aggregated to form the final target virtual object.

[0068] In the above embodiments of the present application, adjusting the blendshape parameters of the initial virtual object based on the 3D model to obtain the adjusted blendshape parameters includes: determining the parameter information of each facial component based on the 3D model; adjusting the blendshape parameters based on the parameter information of each facial component to obtain the adjusted blendshape parameters.

[0069] The 3D model in the above steps can be expressed by a set of continuously adjustable parameters, that is, the 3D model can be expressed by the detailed parameters of the facial features. Therefore, the above parameter information can be facial feature parameters.

[0070] In an alternative embodiment, the 3D model can be expressed by the detailed parameters of different facial components. Therefore, after performing 3D modeling on the face in the photo using the single photo face reconstruction technology based on 3DMM, the detailed parameters of its facial features can be estimated, and the Blendshape value of the initial virtual object can be corrected based on the detailed parameters of the facial features.

[0071] In the above embodiments of the present application, generating an initial virtual object based on multiple facial components in a facial image includes: obtaining the blendshape parameters matching each facial component, where the blendshape parameters are used to characterize the deformation degree of the facial component; aggregating the blendshape parameters to generate an initial virtual object.

[0072] In an alternative embodiment, different classifications can be defined in advance for different facial components included in the facial image, and the corresponding Blendshape values for different classifications can be pre-made to generate a corresponding material library. After identifying each facial component in the facial image, the Blendshape values that match each facial component can be obtained by querying the material library, and the Blendshape values of different facial components can be aggregated into an initial virtual object.

[0073] In the above embodiments of the present application, a multi-task image classification network model is used to process the facial image, identify multiple facial components, and determine the type of each facial component.

[0074] The multi-task image classification network model in the above steps can be a convolutional neural network model, and the backbone network can be selected as resnet50, but it is not limited thereto, and can be determined according to the actual processing accuracy and processing speed. The input of the neural network model can be the facial image, and the output is the predefined facial feature types of the human face.

[0075] The type in the above steps can be the classification results set for different facial components. For example, for the facial component of eyes, the type can be phoenix eyes, peach blossom eyes, standard eyes, etc.; for the facial component of face shape, the type can be ordinary face, square face, round face, etc.

[0076] In an alternative embodiment, a large number of face images are first obtained as training samples, and the facial features are manually labeled for classification, so that a convolutional neural network model can be trained based on the training samples, and a multi-task image classification network model can be obtained. After the multi-task image classification network model is trained, the obtained face image can be input into the multi-task image classification network model to extract the facial feature and obtain the types of multiple facial components.

[0077] In the above embodiments of the present application, obtaining the blend shape parameters matching each facial component includes: based on the type of each facial component, matching the corresponding material for each type from the material library; and determining the blend shape parameters based on the matched material.

[0078] The material library in the above steps may be a database storing Blendshape values corresponding to different facial feature classifications.

[0079] In an alternative embodiment, the multi-task image classification network model can be used to identify the specific classification of each facial component in the face image, and then by querying the material library, the Blendshape value of each classification can be matched, that is, the Blendshape value of each facial component can be matched.

[0080] The following Figure 3 will describe in detail a preferred embodiment of the present application. As Figure 3 shown, this method can be executed by a front-end client or a back-end server, and the method includes the following steps:

[0081] Step S31: Obtain a frontal face photo and preprocess it to obtain the face area, that is, obtain the face image.

[0082] Step S32: Input the image into the multi-task image classification network, and according to the trained network weights, predict the categories to which each part of the input face belongs, including eyes, mouth, face, nose, etc., and select the corresponding Blendshape values of each category from the material library. For example, taking the eyes shown in the Figure 3 square as an example, the input face is classified as a "standard eye".

[0083] Step S33: Use the single-face photo modeling technology based on 3DMM to estimate the three-dimensional model of the face in the photo with the image as the input.

[0084] Step S34: Based on the obtained three-dimensional face model, make further fine-tuning on the basis of the Blendshape values corresponding to each category, so that the Blendshape values finally conform to the actual face appearance.

[0085] Through the above steps, this application proposes a method based on a multi-task image classification network plus 3DMM fine-tuning. Without using expensive depth sensors such as infrared and visible structured light, it can automatically recover various parameters of facial features from a face photo without the need for professional designers, thereby realizing the creation of the face of an automatic 3D virtual avatar. This method avoids the dependence on high-cost equipment, greatly reduces the workload of designers, simplifies the image creation process that is the basis of 3D virtual avatars, and brings better user experience and actual economic and social benefits.

[0086] In addition, by introducing a multi-task image classification network, many interference factors such as illumination, face pose, and occluders in actual photos are greatly reduced; by fine-tuning the Blendshape value according to the 3D model of the face in a single image, on the one hand, the range of changes is limited, so the possibility of incorrect results that do not resemble a human face is greatly reduced. At the same time, after adding the information of the 3D model, the change of the Blendshape value can follow more reasonable constraints, and extreme changes in the Blendshape value can be well avoided, so that the Blendshape value can have stronger uniqueness and difference according to the characteristics of different faces, and the final fine-tuning result can obtain a better subjective evaluation.

[0087] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

[0089] Embodiment 2

[0090] According to an embodiment of the present application, there is also provided a method for processing a face image. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0091] Figure 4 It is a flowchart of a second method for processing a face image according to an embodiment of the present application. As Figure 4 shown, the method may include the following steps:

[0092] Step S402, receiving a face image of an entity object, where the face image includes images of multiple face components of the entity object.

[0093] The entity object in the above steps may be a real object or creature that needs to be virtualized in image. Since the present application is directed to the face virtualization of the entity object, in an embodiment of the present application, the above entity object may be a real person, a robot, etc., but not limited thereto. The face components in the above steps may refer to the five facial features of a human face, including face shape, eyebrows, eyes, nose, mouth, etc., which can be set according to the generation requirements of the virtual image.

[0094] It should be noted that the main body for receiving the face image may be a background server, such as a cloud server, but not limited thereto. The main body for uploading the face image may be a front-end client, that is, an application installed on a mobile terminal such as a mobile phone or a tablet computer used by a user, and may also be a computer terminal such as a notebook computer or a desktop computer, etc., but not limited thereto. The front-end client can communicate with the background server through the Internet, WIFI, 3G, 4G or 5G, etc.

[0095] In an optional embodiment, the user can use a shooting device such as a camera, a camera, a mobile phone, a tablet computer, etc. to shoot the entity object, so as to obtain a face image of the entity object. After obtaining the face image by shooting, the user can operate on the front-end client, select the face image to be processed, and upload it to the back-end server.

[0096] Step S404, generating an initial virtual object based on the multiple face components in the face image.

[0097] Step S406, constructing a three-dimensional model of the face image.

[0098] Step S408, adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0099] Step S410, outputting the target virtual object.

[0100] In an alternative embodiment, after generating the target virtual object, the background server can transmit the target virtual object to the front-end client through the Internet, WIFI, 3G, 4G, 5G, etc., and the front-end client displays it for the user to view.

[0101] In the above embodiments of the present application, adjusting the initial virtual object based on the three-dimensional model to generate the target virtual object includes: adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blend shape parameters; aggregating the adjusted blend shape parameters to generate the target virtual object.

[0102] The blend shape parameters in the above steps can be Blendshape values. The virtual object can be expressed as a combination of Blendshape values of facial features.

[0103] In the above embodiments of the present application, adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blend shape parameters includes: determining the parameter information of each facial component based on the three-dimensional model; adjusting the blend shape parameters based on the parameter information of each facial component to obtain the adjusted blend shape parameters.

[0104] The three-dimensional model in the above steps can be expressed by a set of continuously adjustable parameters, that is, the three-dimensional model can be expressed by the detail parameters of facial features. Therefore, the above parameter information can be facial feature parameters.

[0105] In the above embodiments of the present application, generating the initial virtual object based on multiple facial components in the facial image includes: obtaining the blend shape parameters matching each facial component, where the blend shape parameters are used to characterize the deformation degree of the facial component; aggregating the blend shape parameters to generate the initial virtual object.

[0106] In the above embodiments of the present application, a multi-task image classification network model is used to process the facial image, identify multiple facial components, and determine the type of each facial component.

[0107] The multi-task image classification network model in the above steps can be a convolutional neural network model, and the backbone network can be selected as resnet50, but not limited to this, and can be determined according to the actual processing accuracy and processing speed. The input of the neural network model can be the facial image, and the output is the predefined types of facial features.

[0108] The types in the above steps can be the classification results set for different facial components. For example, for the facial component of eyes, the types can be phoenix eyes, peach blossom eyes, standard eyes, etc.; for the facial component of face shape, the types can be ordinary face, square face, round face, etc.

[0109] In the above embodiments of the present application, obtaining the mixed shape parameters matching each facial component includes: based on the type of each facial component, matching the corresponding materials for each type from the material library; and determining the mixed shape parameters based on the matched materials.

[0110] The material library in the above steps may be a database storing Blendshape values corresponding to different facial feature classifications.

[0111] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0112] Embodiment 3

[0113] According to an embodiment of the present application, there is also provided a method for processing a facial image. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0114] Figure 5 is a flowchart of a third method for processing a facial image according to an embodiment of the present application. As Figure 5 shown, the method may include the following steps:

[0115] Step S502, displaying a facial image of an entity object on an interaction interface, where the facial image includes images of multiple facial components of the entity object.

[0116] The interaction interface in the above steps may be an operation interface provided to the user on the display screen of a device for implementing facial image processing. For example, the interaction interface may be an operation interface displayed on the display screen of a mobile terminal, or may be an operation interface displayed on the display screen of a computer terminal, but is not limited thereto.

[0117] The entity object in the above steps may be a real object or creature that needs to be virtualized in image. Since the present application is directed to the facial virtualization of the entity object, in the embodiments of the present application, the above entity object may be a real person, a robot, etc., but is not limited thereto. The facial components in the above steps may refer to the facial features of a human face, including face shape, eyebrows, eyes, nose, mouth, etc., and can be set according to the needs of generating a virtual image.

[0118] In an optional embodiment, after the facial image is captured, the user can click as Figure 6Select the captured face image by clicking the "Upload Image" button as shown, or by dragging the face image to the dashed box. The face image can be displayed in, for example, Figure 6 the first display area shown.

[0119] In another alternative embodiment, the user can directly capture a face image by clicking the "Capture Image" button as shown in Figure 6 . The captured face image can be displayed in the first display area as shown in Figure 6 .

[0120] Step S504: If an image operation instruction is detected in any area of the interaction interface, trigger the generation of an initial virtual object based on multiple face components in the face image and construct a three-dimensional model of the face image.

[0121] The image operation instruction in the above steps can be an instruction generated by the user clicking a specific button on the interaction interface or an instruction generated by the user performing a predetermined gesture operation on the interaction interface. This instruction is used to generate a corresponding virtual image based on the face image.

[0122] In an alternative embodiment, after uploading the face image, the user can generate an image operation instruction by clicking the "Virtual Image Generation" button as shown in Figure 6 , or directly generate an image operation instruction through a gesture operation. Thus, the computer terminal, mobile terminal, or server can receive this image operation instruction, generate an initial virtual object based on multiple face components in the face image, and construct a three-dimensional model of the face image.

[0123] Step S506: Display the target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

[0124] In an alternative embodiment, the virtual object can be displayed in the second display area as shown in Figure 6 .

[0125] In the above embodiments of the present application, adjusting the initial virtual object based on the three-dimensional model to generate the target virtual object includes: adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blend shape parameters; aggregating the adjusted blend shape parameters to generate the target virtual object.

[0126] The blend shape parameters in the above steps can be Blendshape values. The virtual object can be expressed as a combination of Blendshape values of facial features.

[0127] In the above embodiments of the present application, adjusting the blend shape parameters of the initial virtual object based on the 3D model to obtain the adjusted blend shape parameters includes: determining the parameter information of each facial component based on the 3D model; adjusting the blend shape parameters based on the parameter information of each facial component to obtain the adjusted blend shape parameters.

[0128] The 3D model in the above steps can be expressed by a set of continuously adjustable parameters, that is, the 3D model can be expressed by the detailed parameters of the facial features. Therefore, the above parameter information can be facial feature parameters.

[0129] In the above embodiments of the present application, generating an initial virtual object based on multiple facial components in a facial image includes: obtaining the blend shape parameters matching each facial component, where the blend shape parameters are used to characterize the deformation degree of the facial component; aggregating the blend shape parameters to generate an initial virtual object.

[0130] In the above embodiments of the present application, a multi-task image classification network model is used to process the facial image, identify multiple facial components, and determine the type of each facial component.

[0131] The multi-task image classification network model in the above steps can be a convolutional neural network model, and the backbone network can be selected as resnet50, but it is not limited to this, and it can be determined according to the actual processing accuracy and processing speed. The input of the neural network model can be the facial image, and the output is the predefined types of facial features.

[0132] The type in the above steps can be the classification results set for different facial components. For example, for the facial component of eyes, the types can be phoenix eyes, peach blossom eyes, standard eyes, etc.; for the facial component of face shape, the types can be ordinary face, square face, round face, etc.

[0133] In the above embodiments of the present application, obtaining the blend shape parameters matching each facial component includes: matching the materials corresponding to each type from the material library based on the type of each facial component; determining the blend shape parameters based on the matched materials.

[0134] The material library in the above steps can be a database storing Blendshape values corresponding to different facial feature classifications.

[0135] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0136] Embodiment 4

[0137] According to an embodiment of the present application, there is also provided a method for processing a face image. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0138] Figure 7 is a flowchart of a fourth method for processing a face image according to an embodiment of the present application. As Figure 7 shown, the method may include the following steps:

[0139] Step S702, obtain a face image of an entity object by calling a first interface, where the first interface includes: a first parameter, and the parameter value of the first parameter is the face image, and the face image includes images of multiple face components of the entity object.

[0140] The first interface in the above steps may be an interface for data interaction between a front-end client and a back-end server. The front-end client can pass the face image into an interface function as a parameter of the interface function to achieve the purpose of uploading the face image to the back-end server.

[0141] The entity object in the above steps may be a real object or creature that needs to be virtualized in image. Since the present application is directed to the face virtualization of the entity object, in an embodiment of the present application, the above entity object may be a real person, a robot, etc., but is not limited thereto. The face components in the above steps may refer to the five sense organs of the human face, including face shape, eyebrows, eyes, nose, mouth, etc., and can be set according to the generation requirements of the virtual image.

[0142] Step S704, generate an initial virtual object based on multiple face components in the face image.

[0143] Step S706, construct a three-dimensional model of the face image.

[0144] Step S708, adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0145] Step S710, output the target virtual object by calling a second interface, where the second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

[0146] The second interface in the above steps may be an interface for data interaction between the back-end server and the front-end client. The back-end server can pass the target virtual object into an interface function as a parameter of the interface function to achieve the purpose of sending the target virtual object to the front-end client.

[0147] In the above embodiments of the present application, adjusting the initial virtual object based on the three-dimensional model to generate the target virtual object includes: adjusting the blendshape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blendshape parameters; aggregating the adjusted blendshape parameters to generate the target virtual object.

[0148] The blendshape parameters in the above steps can be Blendshape values. The virtual object can be expressed as a combination of Blendshape values of facial features.

[0149] In the above embodiments of the present application, adjusting the blendshape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blendshape parameters includes: determining the parameter information of each facial component based on the three-dimensional model; adjusting the blendshape parameters based on the parameter information of each facial component to obtain the adjusted blendshape parameters.

[0150] The three-dimensional model in the above steps can be expressed by a set of continuously adjustable parameters, that is, the three-dimensional model can be expressed by the detail parameters of facial features. Therefore, the above parameter information can be facial feature parameters.

[0151] In the above embodiments of the present application, generating the initial virtual object based on multiple facial components in the facial image includes: obtaining the blendshape parameters matching each facial component, where the blendshape parameters are used to characterize the deformation degree of the facial component; aggregating the blendshape parameters to generate the initial virtual object.

[0152] In the above embodiments of the present application, the multi-task image classification network model is used to process the facial image, identify multiple facial components, and determine the type of each facial component.

[0153] The multi-task image classification network model in the above steps can be a convolutional neural network model, and the backbone network can be selected as resnet50, but not limited to this, and can be determined according to the actual processing accuracy and processing speed. The input of the neural network model can be the facial image, and the output is the predefined facial feature types.

[0154] The type in the above steps can be the classification result set for different facial components. For example, for the facial component of eyes, the type can be phoenix eyes, peach blossom eyes, standard eyes, etc.; for the facial component of face shape, the type can be ordinary face, square face, round face, etc.

[0155] In the above embodiments of the present application, obtaining the blendshape parameters matching each facial component includes: based on the type of each facial component, matching the corresponding material for each type from the material library; determining the blendshape parameters based on the matched material.

[0156] The material library in the above steps can be a database storing Blendshape values corresponding to different facial feature classifications.

[0157] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as those provided in Embodiment 1 in terms of the scheme, application scenario, and implementation process, but are not limited to the scheme provided in Embodiment 1.

[0158] Embodiment 5

[0159] According to an embodiment of the present application, there is also provided a facial image processing device for implementing the above facial image processing method, as Figure 8 shown. The device 800 includes: an acquisition module 802, a first generation module 804, a construction module 806, and a second generation module 808.

[0160] Among them, the acquisition module 802 is used to acquire a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; the first generation module 804 is used to generate an initial virtual object based on the multiple facial components in the facial image; the construction module 806 is used to construct a three-dimensional model of the facial image; the second generation module 808 is used to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0161] It should be noted here that the above acquisition module 802, first generation module 804, construction module 806, and second generation module 808 correspond to steps S202 to S208 in Embodiment 1. The instances and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0162] In the above embodiments of the present application, the second generation module includes: an adjustment unit and a first generation unit.

[0163] Among them, the adjustment unit is used to adjust the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted blend shape parameters; the first generation unit is used to aggregate the adjusted blend shape parameters to generate a target virtual object.

[0164] In the above embodiments of the present application, the adjustment unit includes: a first determination subunit and an adjustment subunit.

[0165] Among them, the first determination subunit is used to determine the parameter information of each facial component based on the three-dimensional model; the adjustment subunit is used to adjust the blend shape parameters based on the parameter information of each facial component to obtain adjusted blend shape parameters.

[0166] In the above embodiments of the present application, the first generation module includes: an acquisition unit and a second generation unit.

[0167] Among them, the acquisition unit is used to acquire the blend shape parameters matching each facial component, where the blend shape parameters are used to characterize the deformation degree of the facial component; the second generation unit is used to aggregate the blend shape parameters to generate an initial virtual object.

[0168] In the above embodiments of the present application, the device further includes: a processing module.

[0169] Among them, the processing module is used to process the facial image by using a multi-task image classification network model, identify multiple facial components, and determine the type of each facial component.

[0170] In the above embodiments of the present application, the acquisition unit includes: a matching subunit and a second determination subunit.

[0171] Among them, the matching subunit is used to match the materials corresponding to each type from the material library based on the type of each facial component; the second determination subunit is used to determine the blend shape parameters based on the matched materials.

[0172] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0173] Embodiment 6

[0174] According to the embodiments of the present application, there is also provided a facial image processing device for implementing the above facial image processing method, as Figure 9 shown, the device 900 includes: a receiving module 902, a first generation module 904, a construction module 906, a second generation module 908, and an output module 910.

[0175] Among them, the receiving module 902 is used to receive the facial image of the entity object, where the facial image includes the images of multiple facial components of the entity object; the first generation module 904 is used to generate an initial virtual object based on the multiple facial components in the facial image; the construction module 906 is used to construct a three-dimensional model of the facial image; the second generation module 908 is used to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; the output module 910 is used to output the target virtual object.

[0176] It should be noted here that the above receiving module 902, first generating module 904, constructing module 906, second generating module 908, and output module 910 correspond to steps S402 to S410 in Embodiment 2. The instances and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0177] In the above embodiments of the present application, the second generating module includes: an adjustment unit and a first generating unit.

[0178] Among them, the adjustment unit is used to adjust the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blend shape parameters; the first generating unit is used to aggregate the adjusted blend shape parameters to generate a target virtual object.

[0179] In the above embodiments of the present application, the adjustment unit includes: a first determination subunit and an adjustment subunit.

[0180] Among them, the first determination subunit is used to determine the parameter information of each face component based on the three-dimensional model; the adjustment subunit is used to adjust the blend shape parameters based on the parameter information of each face component to obtain the adjusted blend shape parameters.

[0181] In the above embodiments of the present application, the first generating module includes: an acquisition unit and a second generating unit.

[0182] Among them, the acquisition unit is used to acquire the blend shape parameters matching each face component, where the blend shape parameters are used to characterize the deformation degree of the face component; the second generating unit is used to aggregate the blend shape parameters to generate an initial virtual object.

[0183] In the above embodiments of the present application, the device further includes: a processing module.

[0184] Among them, the processing module is used to process the face image by using a multi-task image classification network model, identify multiple face components, and determine the type of each face component.

[0185] In the above embodiments of the present application, the acquisition unit includes: a matching subunit and a second determination subunit.

[0186] Among them, the matching subunit is used to match the materials corresponding to each type from the material library based on the type of each face component; the second determination subunit is used to determine the blend shape parameters based on the matched materials.

[0187] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0188] Embodiment 7

[0189] According to an embodiment of the present application, there is also provided a facial image processing device for implementing the above facial image processing method. As Figure 10 shown, the device 1000 includes: a first display module 1002, a trigger module 1004, and a second display module 1006.

[0190] Among them, the first display module 1002 is used to display the facial image of the entity object on the interaction interface, where the facial image includes the images of multiple facial components of the entity object; the trigger module 1004 is used to trigger the generation of an initial virtual object based on the multiple facial components in the facial image and construct a three-dimensional model of the facial image if an image operation instruction is detected in any area of the interaction interface; the second display module 1006 is used to display the target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

[0191] It should be noted here that the above first display module 1002, trigger module 1004, and second display module 1006 correspond to steps S502 to S506 in Embodiment 4. The instances and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0192] In the above embodiments of the present application, the device further includes: an adjustment module and a generation module.

[0193] Among them, the adjustment module is used to adjust the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blend shape parameters; the generation module is used to aggregate the adjusted blend shape parameters to generate the target virtual object.

[0194] In the above embodiments of the present application, the adjustment module includes: a determination unit and an adjustment unit.

[0195] Among them, the determination unit is used to determine the parameter information of each facial component based on the three-dimensional model; the adjustment unit is used to adjust the blend shape parameters based on the parameter information of each facial component to obtain the adjusted blend shape parameters.

[0196] In the above embodiments of the present application, the trigger module includes: an acquisition unit and a generation unit.

[0197] Among them, the obtaining unit is used to obtain blend shape parameters matching each facial component, where the blend shape parameters are used to characterize the deformation degree of the facial component; the generating unit is used to aggregate the blend shape parameters to generate an initial virtual object.

[0198] In the above embodiment of the present application, the device further includes: a processing module.

[0199] Among them, the processing module is used to process the facial image by using a multi-task image classification network model, identify multiple facial components, and determine the type of each facial component.

[0200] In the above embodiment of the present application, the obtaining unit includes: a matching subunit and a determining subunit.

[0201] Among them, the matching subunit is used to match the materials corresponding to each type from the material library based on the type of each facial component; the determining subunit is used to determine the blend shape parameters based on the matched materials.

[0202] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0203] Embodiment 8

[0204] According to an embodiment of the present application, there is also provided a facial image processing device for implementing the above facial image processing method. As Figure 11 shown, the device 1100 includes: a first calling module 1102, a first generating module 1104, a building module 1106, a second generating module 1108, and a second calling module 1110.

[0205] Among them, the first calling module 1102 is used to obtain the facial image of the entity object by calling the first interface. The first interface includes: a first parameter, and the parameter value of the first parameter is the facial image, and the facial image includes the images of multiple facial components of the entity object; the first generating module 1104 is used to generate an initial virtual object based on the multiple facial components in the facial image; the building module 1106 is used to build a three-dimensional model of the facial image; the second generating module 1108 is used to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; the second calling module 1110 is used to output the target virtual object by calling the second interface. The second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

[0206] It should be noted here that the above first calling module 1102, first generating module 1104, constructing module 1106, second generating module 1108, and second calling module 1110 correspond to steps S702 to S710 in Embodiment 4. The instances and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0207] In the above embodiments of the present application, the second generating module includes: an adjustment unit and a first generating unit.

[0208] Among them, the adjustment unit is used to adjust the blend shape parameters of the initial virtual object based on the 3D model to obtain the adjusted blend shape parameters; the first generating unit is used to aggregate the adjusted blend shape parameters to generate a target virtual object.

[0209] In the above embodiments of the present application, the adjustment unit includes: a first determination subunit and an adjustment subunit.

[0210] Among them, the first determination subunit is used to determine the parameter information of each face component based on the 3D model; the adjustment subunit is used to adjust the blend shape parameters based on the parameter information of each face component to obtain the adjusted blend shape parameters.

[0211] In the above embodiments of the present application, the first generating module includes: an acquisition unit and a second generating unit.

[0212] Among them, the acquisition unit is used to acquire the blend shape parameters matching each face component, where the blend shape parameters are used to characterize the deformation degree of the face component; the second generating unit is used to aggregate the blend shape parameters to generate an initial virtual object.

[0213] In the above embodiments of the present application, the device further includes: a processing module.

[0214] Among them, the processing module is used to process the face image using a multi-task image classification network model, identify multiple face components, and determine the type of each face component.

[0215] In the above embodiments of the present application, the acquisition unit includes: a matching subunit and a second determination subunit.

[0216] Among them, the matching subunit is used to match the materials corresponding to each type from the material library based on the type of each face component; the second determination subunit is used to determine the blend shape parameters based on the matched materials.

[0217] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as those provided in Embodiment 1 in terms of the scheme, application scenario, and implementation process, but are not limited to the scheme provided in Embodiment 1.

[0218] Embodiment 9

[0219] According to an embodiment of the present application, there is also provided a processing system for facial images, including:

[0220] a processor; and

[0221] a memory, connected to the processor, for providing instructions for the processor to perform the following processing steps: obtaining a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; and adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0222] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as those provided in Embodiment 1 in terms of the scheme, application scenario, and implementation process, but are not limited to the scheme provided in Embodiment 1.

[0223] Embodiment 10

[0224] According to an embodiment of the present application, there is also provided a method for processing facial images. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0225] Figure 12 is a flowchart of a fifth method for processing facial images according to an embodiment of the present application. As Figure 12 shown, the method may include the following steps:

[0226] Step S1202, when a communication request is received, obtaining a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object.

[0227] The communication request in the above step may be a video call request initiated between users. Users can initiate a communication request through mobile terminals such as smartphones, tablets, laptops, AR (Augmented Reality) devices, or VR (Virtual Reality) devices. After the communication request is initiated, the real image or virtual image of the user can be displayed in the call interface. In the embodiments of the present application, a virtual facial image is taken as an example for illustration.

[0228] The entity object in the above steps can be a real object or creature that needs to be visually virtualized. Since this application focuses on the facial virtualization of entity objects, in the embodiments of this application, the above entity object can be a real person, a robot, etc., but is not limited thereto. The facial components in the above steps can refer to the five facial features, including face shape, eyebrows, eyes, nose, mouth, etc., and can be set according to the generation requirements of the virtual image.

[0229] Step S1204: Generate an initial virtual object based on multiple facial components in the facial image.

[0230] Step S1206: Construct a three-dimensional model of the facial image.

[0231] Step S1208: Adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0232] Step S1210: Display the target virtual object in the communication interface.

[0233] The communication interface in the above steps can be a video call interface, and the images of both or multiple parties in the call, including real images and virtual images, can be displayed in this interface.

[0234] For example, taking VR device communication as an example, when the VR device of user A receives a video call request initiated by the VR device of user B, if user A determines to have a video call with user B and needs to display a virtual image during the call, user A's facial image can be collected through the VR device, and an initial virtual image can be generated based on different facial components in the facial image. Further, a three-dimensional model of user A's face can be constructed, and the initial virtual image can be adjusted based on the three-dimensional model to obtain the virtual image finally shown to user B for viewing. This virtual image is displayed in the video call interface, and both user A and user B can view it.

[0235] In the above embodiments of this application, adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object includes: adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted blend shape parameters; aggregating the adjusted blend shape parameters to generate a target virtual object.

[0236] The blend shape parameters in the above steps can be Blendshape values. The virtual object can be expressed as a combination of Blendshape values of the five facial features.

[0237] In the above embodiments of the present application, adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain the adjusted blend shape parameters includes: determining the parameter information of each facial component based on the three-dimensional model; adjusting the blend shape parameters based on the parameter information of each facial component to obtain the adjusted blend shape parameters.

[0238] The three-dimensional model in the above steps can be expressed by a set of continuously adjustable parameters, that is, the three-dimensional model can be expressed by the detailed parameters of the facial features. Therefore, the above parameter information can be facial feature parameters.

[0239] In the above embodiments of the present application, generating an initial virtual object based on multiple facial components in a facial image includes: obtaining the blend shape parameters matching each facial component, where the blend shape parameters are used to characterize the deformation degree of the facial component; aggregating the blend shape parameters to generate an initial virtual object.

[0240] In the above embodiments of the present application, a multi-task image classification network model is used to process the facial image, identify multiple facial components, and determine the type of each facial component.

[0241] The multi-task image classification network model in the above steps can be a convolutional neural network model, and the backbone network can be selected as resnet50, but it is not limited to this, and it can be determined according to the actual processing accuracy and processing speed. The input of the neural network model can be a facial image, and the output is a predefined facial feature type.

[0242] The type in the above steps can be a classification result set for different facial components. For example, for the facial component of eyes, the type can be phoenix eyes, peach blossom eyes, standard eyes, etc.; for the facial component of face shape, the type can be ordinary face, square face, round face, etc.

[0243] In the above embodiments of the present application, obtaining the blend shape parameters matching each facial component includes: matching the materials corresponding to each type from the material library based on the type of each facial component; determining the blend shape parameters based on the matched materials.

[0244] The material library in the above steps can be a database storing Blendshape values corresponding to different facial feature classifications.

[0245] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0246] Embodiment 11

[0247] Embodiments of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above computer terminal may also be replaced with a terminal device such as a mobile terminal.

[0248] Optionally, in this embodiment, the above computer terminal may be located in at least one of multiple network devices in a computer network.

[0249] In this embodiment, the above computer terminal may execute the program code of the following steps in the face image processing method: obtaining a face image of an entity object, where the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image; and adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0250] Optionally, Figure 13 is a structural block diagram of a computer terminal according to an embodiment of the present application. As Figure 13 shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1302, and a memory 1304.

[0251] Among them, the memory may be used to store software programs and modules, such as the program instructions / modules corresponding to the face image processing method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above face image processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely set relative to the processor, and these remote memories may be connected to the terminal A through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0252] The processor may call the information and application programs stored in the memory through a transmission device to execute the following steps: obtaining a face image of an entity object, where the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image; and adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0253] Optionally, the above processor may also execute the program code of the following steps: adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted blend shape parameters; and aggregating the adjusted blend shape parameters to generate a target virtual object.

[0254] Optionally, the above-mentioned processor may also execute program code for the following steps: determining parameter information of each facial component based on the three-dimensional model; adjusting the blend shape parameters based on the parameter information of each facial component to obtain the adjusted blend shape parameters.

[0255] Optionally, the above-mentioned processor may also execute program code for the following steps: obtaining blend shape parameters matching each facial component, where the blend shape parameters are used to characterize the deformation degree of the facial component; aggregating the blend shape parameters to generate an initial virtual object.

[0256] Optionally, the above-mentioned processor may also execute program code for the following steps: processing the facial image using a multi-task image classification network model to identify multiple facial components and determine the type of each facial component.

[0257] Optionally, the above-mentioned processor may also execute program code for the following steps: matching materials corresponding to each type from the material library based on the type of each facial component; determining the blend shape parameters based on the matched materials.

[0258] The processor may call the information and application programs stored in the memory through the transmission device to execute the following steps: receiving a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; outputting the target virtual object.

[0259] The processor may call the information and application programs stored in the memory through the transmission device to execute the following steps: displaying a facial image of an entity object on the interaction interface, where the facial image includes images of multiple facial components of the entity object; if an image operation instruction is detected in any area of the interaction interface, triggering the generation of an initial virtual object based on the multiple facial components in the facial image and constructing a three-dimensional model of the facial image; displaying a target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

[0260] The processor can call the information and application programs stored in the memory through a transmission device to perform the following steps: obtain the facial image of an entity object by calling a first interface, where the first interface includes: a first parameter, and the parameter value of the first parameter is the facial image, and the facial image includes images of multiple facial components of the entity object; generate an initial virtual object based on the multiple facial components in the facial image; construct a three-dimensional model of the facial image; adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; output the target virtual object by calling a second interface, where the second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

[0261] The processor can call the information and application programs stored in the memory through a transmission device to perform the following steps: when receiving a communication request, obtain the facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; generate an initial virtual object based on the multiple facial components in the facial image; construct a three-dimensional model of the facial image; adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; display the target virtual object in a communication interface.

[0262] By adopting the embodiment of the present application, a facial image processing solution is provided. An initial virtual object is generated through multiple facial components in the facial image, and the initial virtual object is finely adjusted based on the constructed three-dimensional model, which can be automatically completed without the manual operation of professional designers, avoiding completely different processing results for the same facial image due to different designers, achieving the technical effects of saving processing time and labor costs, improving the processing effect, and enhancing the user experience and favorability, and further solving the technical problem in the related art that generating virtual objects manually results in relatively high processing costs and poor effects.

[0263] Those of ordinary skill in the art can understand that Figure 13 The structure shown is only schematic, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a mobile Internet device (Mobile Internet Devices, MID), a PAD and other terminal devices. Figure 13 It does not limit the structure of the above-mentioned electronic device. For example, computer terminal A may further include more or fewer components (such as a network interface, a display device, etc.) than those shown Figure 13 in the figure, or have a different configuration from that shown Figure 13 in the figure.

[0264] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, and the storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0265] Embodiment 12

[0266] An embodiment of the present application also provides a storage medium. Optionally, in this embodiment, the above storage medium can be used to store the program code executed by the method for processing the face image provided in the above embodiment.

[0267] Optionally, in this embodiment, the above storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0268] Optionally, in this embodiment, the storage medium is set to store program code for performing the following steps: obtaining a face image of an entity object, where the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image; and adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

[0269] Optionally, the above storage medium is further set to store program code for performing the following steps: adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted blend shape parameters; and aggregating the adjusted blend shape parameters to generate a target virtual object.

[0270] Optionally, the above storage medium is further set to store program code for performing the following steps: determining parameter information of each face component based on the three-dimensional model; and adjusting the blend shape parameters based on the parameter information of each face component to obtain adjusted blend shape parameters.

[0271] Optionally, the above storage medium is further set to store program code for performing the following steps: obtaining blend shape parameters matching each face component, where the blend shape parameters are used to characterize the deformation degree of the face component; and aggregating the blend shape parameters to generate an initial virtual object.

[0272] Optionally, the above storage medium is further set to store program code for performing the following steps: processing the face image using a multi-task image classification network model to identify multiple face components and determine the type of each face component.

[0273] Optionally, the above storage medium is further configured to store program code for performing the following steps: matching, from a material library, materials corresponding to each type based on the type of each facial component; and determining blend shape parameters based on the matched materials.

[0274] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a facial image of an entity object, where the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; and outputting the target virtual object.

[0275] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: displaying a facial image of an entity object on an interaction interface, where the facial image includes images of multiple facial components of the entity object; if an image operation instruction is detected in any area of the interaction interface, triggering the generation of an initial virtual object based on the multiple facial components in the facial image and constructing a three-dimensional model of the facial image; and displaying a target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

[0276] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a facial image of an entity object by calling a first interface, where the first interface includes: a first parameter, and the parameter value of the first parameter is the facial image, and the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; and outputting the target virtual object by calling a second interface, where the second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

[0277] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a facial image of an entity object when a communication request is received, where the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; and presenting the target virtual object in a communication interface.

[0278] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments.

[0279] In the above embodiments of the present application, the descriptions of the various embodiments each have their own emphasis. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0280] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0281] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0282] In addition, the functional units in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0283] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs, etc., which can store program codes.

[0284] The above is only the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for processing a facial image, characterized in that, including: obtaining a face image of an entity object, where the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

2. The method according to claim 1, wherein Adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object includes: adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted blend shape parameters, where the blend shape parameters are used to characterize the deformation degree of the face components; aggregating the adjusted blend shape parameters to generate the target virtual object.

3. The method according to claim 2, wherein Adjusting the blend shape parameters of the initial virtual object based on the three-dimensional model to obtain adjusted blend shape parameters includes: determining parameter information of each face component based on the three-dimensional model; adjusting the blend shape parameters based on the parameter information of each face component to obtain the adjusted blend shape parameters.

4. The method according to claim 1, wherein constructing a three-dimensional model of the face image using a three-dimensional deformable model.

5. The method according to claim 1, wherein Generating an initial virtual object based on the multiple face components in the face image includes: obtaining blend shape parameters matching each face component, where the blend shape parameters are used to characterize the deformation degree of the face components; aggregating the blend shape parameters to generate the initial virtual object.

6. The method according to claim 5, wherein processing the face image using a multi-task image classification network model to identify the multiple face components and determine the type of each face component.

7. The method according to claim 6, characterized in that, Obtaining blend shape parameters matching each face component includes: matching materials corresponding to each type from a material library based on the type of each face component; determining the blend shape parameters based on the matched materials.

8. A method for processing a facial image, characterized in that, including: receiving a face image of an entity object, where the face image includes images of multiple face components of the entity object; generating an initial virtual object based on the multiple face components in the face image; constructing a three-dimensional model of the face image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; outputting the target virtual object.

9. A method for processing a facial image, characterized in that, including: displaying a face image of an entity object on an interaction interface, where the face image includes images of multiple face components of the entity object; if an image operation instruction is detected in any area of the interaction interface, triggering the generation of an initial virtual object based on the multiple face components in the face image and constructing a three-dimensional model of the face image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; displaying a target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

10. A method for processing a facial image, characterized in that, including: Obtain the facial image of the entity object by calling the first interface, where the first interface includes: a first parameter, and the parameter value of the first parameter is the facial image, and the facial image includes images of multiple facial components of the entity object; Generate an initial virtual object based on the multiple facial components in the facial image; Construct a three-dimensional model of the facial image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; Adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; Output the target virtual object by calling the second interface, where the second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

11. A processing device for facial images, characterized in that, Include: An acquisition module, configured to obtain the facial image of the entity object, where the facial image includes images of multiple facial components of the entity object; A first generation module, configured to generate an initial virtual object based on the multiple facial components in the facial image; A construction module, configured to construct a three-dimensional model of the facial image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; A second generation module, configured to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object.

12. A processing device for facial images, characterized in that, Include: A receiving module, configured to receive the facial image of the entity object, where the facial image includes images of multiple facial components of the entity object; A first generation module, configured to generate an initial virtual object based on the multiple facial components in the facial image; A construction module, configured to construct a three-dimensional model of the facial image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; A second generation module, configured to adjust the initial virtual object based on the three-dimensional model to generate a target virtual object; An output module, configured to output the target virtual object.

13. A processing device for facial images, characterized in that, Include: A first display module, configured to display the facial image of the entity object on the interaction interface, where the facial image includes images of multiple facial components of the entity object; A trigger module, configured to, if an image operation instruction is detected in any area of the interaction interface, trigger the generation of an initial virtual object based on the multiple facial components in the facial image, and construct a three-dimensional model of the facial image, where the three-dimensional model is expressed by the facial feature parameters of the entity object; A second display module, configured to display the target virtual object on the interaction interface, where the target virtual object is generated by adjusting the initial virtual object based on the three-dimensional model.

14. A processing device for facial images, characterized in that, Include: A first call module, configured to obtain the facial image of the entity object by calling the first interface, where the first interface includes: a first parameter, and the parameter value of the first parameter is the facial image, and the facial image includes images of multiple facial components of the entity object; A first generation module, configured to generate an initial virtual object based on the multiple facial components in the facial image; A construction module for constructing a three-dimensional model of the facial image, wherein the three-dimensional model is expressed by the facial feature parameters of the entity object; A second generation module for adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; A second call module for outputting the target virtual object by calling a second interface, wherein the second interface includes: a second parameter, and the parameter value of the second parameter is the target virtual object.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the facial image processing method according to any one of claims 1 to 10.

16. A processing terminal, characterized in that, Comprising: A memory and a processor, the processor is used to run the program stored in the memory, wherein when the program runs, it executes the facial image processing method according to any one of claims 1 to 10.

17. A processing system for facial images, characterized in that, Comprising: A processor; And A memory, connected to the processor, for providing instructions for the processor to perform the following processing steps: obtaining a facial image of an entity object, wherein the facial image includes images of multiple facial components of the entity object; generating an initial virtual object based on the multiple facial components in the facial image; constructing a three-dimensional model of the facial image, wherein the three-dimensional model is expressed by the facial feature parameters of the entity object; adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object.

18. A method for processing a facial image, characterized in that, Comprising: When receiving a communication request, obtaining a facial image of an entity object, wherein the facial image includes images of multiple facial components of the entity object; Generating an initial virtual object based on the multiple facial components in the facial image; Constructing a three-dimensional model of the facial image, wherein the three-dimensional model is expressed by the facial feature parameters of the entity object; Adjusting the initial virtual object based on the three-dimensional model to generate a target virtual object; Displaying the target virtual object in a communication interface.

Images