A virtual character face pinching method and system, a storage medium and a processing terminal

Abstract

The present application relates to the field of virtual character generation, and particularly relates to a virtual character face pinching method and system, a storage medium and a processing terminal, the method comprising: obtaining a face picture of a person to be generated as the virtual character; preprocessing the face picture to obtain a model input picture through face detection and face alignment; inputting the model input picture into a gender classification model for gender recognition; selecting a corresponding face pinching inference module according to the gender recognition result, and then obtaining a virtual image code; and generating a customized virtual image according to the virtual image code. In the backbone network in the face pinching inference module, a series of lightweight network modules are used as feature extraction networks to accelerate the convergence of the network and improve the performance of the network, and the output head adopts a feature sharing structure to make the subtasks with correlation promote each other, thereby improving the output precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of virtual character generation, and in particular to a virtual character face sculpting method and system, a storage medium and a processing terminal. BACKGROUND

[0002] Most of the existing methods for creating and customizing game characters currently require players to manually adjust the facial features of the characters in order to recreate their own faces or sculpt the faces of others. It usually takes a player several hours of patience to manually adjust hundreds of parameters (such as face shape, eyes) to create a character that resembles a specified portrait. This is time-consuming and inefficient. Some developers have also tried to develop new methods to automatically customize the faces of characters by analyzing real person facial images. However, the actual effect of these methods is usually not good, either because the process is complex or because the similarity between the generated virtual character image and the input image is poor, making it difficult to maintain consistency in form and expression with the input image, which is not conducive to improving user experience. SUMMARY

[0003] In view of the above deficiencies of the prior art, the present application aims to provide a virtual character face sculpting method, system, storage medium and processing terminal.

[0004] In one embodiment of the present application, a new virtual character face sculpting method is provided, which comprises:

[0005] obtaining a face picture of a person to be generated as the virtual character; pre-processing the face picture to obtain a model input image through face detection and face alignment; inputting the model input image into a gender classification model for gender recognition; selecting a corresponding face sculpting inference module according to the gender recognition result to obtain a virtual image code; and generating a customized virtual image according to the virtual image code.

[0006] In another embodiment of the present application, the face sculpting inference module is divided into two types for men and women, and the network model mainly includes a backbone network for extracting image features and an output head for outputting corresponding virtual image codes.

[0007] In yet another embodiment of the present application, the backbone network uses a series of lightweight network modules, including a CBA3*3 module, a first Ghost Bottlenecks module, a MobileVit Block, a second Ghost Bottlenecks module and a third Ghost Bottlenecks module.

[0008] In another embodiment of the application, the CBA module is a convolutional network module, comprising a series of connected conv layers, BN layers and activation layers Act; each Ghost Bottlenecks module comprises, in sequence, a depth separable convolution DSC, a Ghost module, a Depthwise Convolution, a BN layer, an Act layer, an SE Block, a BN layer, and the input of the Ghost Bottlenecks module is added to the output after the operation of the Ghost Bottlenecks module to obtain the final output of the Ghost Bottlenecks module.

[0009] In another embodiment of the application, the MobileVit Block comprises, in sequence, a DSC layer, a reshape & Transpose, a Transformer layer, a reshape & Transpose, a conv1*1 layer, the output after the 1*1 convolution is concatenated with the original input of the MobileVit Block, and the output is obtained after the CBA layer.

[0010] In another embodiment of the application, the Transformer layer comprises the following parts: a first LN layer, a multi-head attention layer, a second LN layer, a first FC layer, a first Act layer, a first Dropout layer, a second FC layer, and a second Dropout layer, wherein the output after the multi-head attention layer is added to the input before the first LN layer in a residual manner, and then the output after the second Dropout layer is added to the input before the second LN layer in a residual manner to obtain the final output.

[0011] In another embodiment of the application, the output head adopts a feature sharing structure, and outputs the loss function L total of multiple tasks.

[0012]

[0013] During optimization, ω i is set as:

[0014]

[0015] wherein ω i is the weight of the i-th task, L i is the loss function of the i-th task, N is the number of tasks, T is a constant, the larger T is, the more uniform the loss weight of each task is, r n (t-1) is the loss at two consecutive time points t-1 and t-2, i.e., L nThe result of the division is obtained by adjusting the weight ω of each task i Achieve similar learning rate of each task.

[0016] In an embodiment of the present application, a new virtual character face shaping processing system is provided, comprising a processor and a memory connected to the processor, for providing the processor with instructions to process the following processing steps: obtaining a face picture of the virtual character to be generated; pre-processing the face picture to obtain a model input picture through face detection and face alignment; inputting the model input picture into a gender classification model for gender recognition; selecting a corresponding face shaping inference module according to the gender recognition result, and then obtaining a virtual image code; and generating a customized virtual image according to the virtual image code.

[0017] In an embodiment of the present application, a computer readable storage medium is provided, which comprises a stored computer program, wherein the program controls the device where the computer readable storage medium is located to execute the virtual character face shaping method described above when the program is running.

[0018] In an embodiment of the present application, a processing terminal is provided, characterized by comprising a storage medium and a processor, wherein the processor is used to run a computer program stored in the storage medium, and the program executes the virtual character face shaping method described above when the program is running.

[0019] The present application improves the recognition accuracy by pre-processing the picture through face detection and face alignment, classifying and managing difficult example data, and performing data enhancement processing; the gender recognition is performed in advance before the image is input into the face shaping inference module, which can improve the accuracy of the virtual image on the one hand, and can simplify the overall structure of the module and improve the running efficiency on the other hand by dividing the face shaping inference module into two types of male and female; the backbone network in the face shaping inference module uses a series of lightweight network modules as the feature extraction network to accelerate the convergence of the network and improve the performance of the network; the MobileVit Block is added in the multiple Ghost Bottlenecks modules, which can obtain local and global representation information in each feature map with fewer parameters, and the expression of the features is more accurate through mutual "reference" between the features; the output head adopts a feature sharing structure, which can promote the interaction between the sub-tasks with correlation, and to eliminate the competitive relationship between multiple tasks, the training rate of each task is adjusted to be the same, so as to automatically balance the weight of the multi-task loss function, and then improve the output accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of the new virtual character face shaping method of the present application;

[0021] Figure 2 Fig. 1 is a schematic diagram of a network structure of a face kneading reasoning module according to an embodiment of the present disclosure;

[0022] Figure 3 Fig. 2 is a schematic diagram of a network backbone in the face kneading reasoning module according to an embodiment of the present disclosure;

[0023] Figure 4 Fig. 3 is a schematic diagram of a Ghost Bottlenecks module in the network backbone according to an embodiment of the present disclosure;

[0024] Figure 5 Fig. 4 is a schematic diagram of a MobileVit Block module in the network backbone according to an embodiment of the present disclosure;

[0025] Figure 6 Fig. 5 is a schematic diagram of an output head module in the face kneading reasoning module according to an embodiment of the present disclosure;

[0026] Figure 7 Fig. 6 is a process diagram of a virtual image output according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather should be construed to encompass all modifications equivalent in meaning and scope. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of the present disclosure.

[0028] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions will be given in the description below.

[0030] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] Before detailing the specific embodiments of the present application, first the application scenario of the present application is described as follows: the method for virtual character face pinching provided by the embodiments of the present application can be mainly applied to the field of game technology, or other related technical fields with face pinching requirements. Taking the field of game technology as an example, in the case that a game user intends to use a virtual character for game testing, it is often desired that the virtual character used is more similar to a real person, in order to further improve the game immersion, for example, generating a game character image in a game system through a real face image, using the game character image as a game character in the game system, thereby increasing the user's sense of integration with the game content, so as to achieve the effect of improving the user experience, the face image can be the user's own image, or a user-selected image of others, the game character image can be a three-dimensional image, or a two-dimensional image.

[0034] The technical solutions of the present disclosure will be described in detail below in combination with specific embodiments.

[0035] Figure 1 is a flowchart of a new virtual character face pinching method according to an exemplary embodiment of the present disclosure; see Figure 1 The method comprises the following steps:

[0036] S1: obtaining a face picture of a virtual character to be generated;

[0037] In this step, there are two ways to obtain the face picture, one is to import an existing face image, and the other is to collect in real time; the face picture can be the user's own picture, or a user-selected image of others.

[0038] S2: pre-processing the face picture to obtain a model input image through face detection and face alignment;

[0039] In this step, the purpose of preprocessing of the face picture is to make further processing on the face picture on the basis of detection of the system, that is, gray scale adjustment, image filtering, image size normalization, etc., to facilitate feature extraction of the face picture. Further, some difficult examples are subdivided on the data set, especially the data of extreme cases such as dark facial light, low image quality, and large face angle are classified and managed, data augmentation is performed according to the classification result, and one or more of the following operations are performed: left-right flip transformation operation, random affine transformation operation, transpose transformation operation, non-rigid transformation operation, color random transformation operation, saturation random transformation operation, brightness random transformation operation, contrast random transformation operation, and random erasing operation. Specifically, the main preprocessing operations include:

[0040] Face detection: After the face picture is input, the position and size of the face in the image are accurately marked, and the position of the key points of the face (such as the center of the eyes or the corners of the mouth) is located, and then useful image information (such as histogram features, color features, template features, structure features, and Haar features) is selected while other redundant image information is automatically removed to further ensure accurate collection of face images.

[0041] Face alignment: Geometric correction is performed on the face according to the face position, size, and key point information, that is, the face is changed to a relatively standard size and position through scaling, rotation, stretching, and other image changes.

[0042] S3: inputting the model into a gender classification model for gender recognition;

[0043] In this step, gender recognition is performed before the image is input into the face pinching inference module, which can improve the accuracy of the virtual image and divide the face pinching inference module into two types of male and female, simplifying the overall structure of the module and improving the running efficiency.

[0044] S4: selecting the corresponding face pinching inference module according to the gender recognition result to obtain a virtual image code;

[0045] As shown in Figure 2 In this step, the network model in the face pinching inference module mainly includes a backbone network for extracting image features and a head for outputting a corresponding virtual image code. The model input image after face alignment is input into the face pinching inference module as an input picture, which contains information such as hairstyle, facial features (ears, eyebrows, eyes, nose, and lips), face shape, and gender. The input picture is feature-extracted by the backbone network, and then a corresponding virtual image code is output by the head.

[0046] Further, the backbone extracts features of the input picture through multiple convolutional layers. In order to accelerate the convergence of the network and improve the performance of the network, a series of lightweight network modules are used as the structure of the feature extraction network. In a specific embodiment, as shown in Figure 3 The backbone includes a CBA 3*3 module, a first Ghost Bottlenecks module, a MobileVit Block, a second Ghost Bottlenecks module, and a third Ghost Bottlenecks module.

[0047] The CBA module is a convolutional network module, which includes a Conv layer, a BN layer, and an Act layer connected in series. CBA represents Conv+BN+Act. The Conv layer is used for feature extraction, which needs a filter to traverse the entire image. The filter itself can be regarded as a mathematical matrix. Optionally, the filter can use a Gaussian filter. The BN layer is used for batch normalization processing of the feature extraction result of the Conv layer. That is, when the model is trained by stochastic gradient descent each time, the corresponding response is normalized to make the mean of the output result 0 and the variance 1. This not only accelerates the model convergence speed, but more importantly, it can alleviate the "gradient dispersion" problem of the deep network to some extent, so that the neural network model obtained by training is more stable. The activation layer is activated by introducing nonlinear features, so that the model can cope with more complex data learning or simulation conditions, and improve the learning ability of the model.

[0048] The Ghost Bottleneck obtains "redundant" feature maps through "cheap" operations (DepthwiseConv) to speed up the inference of the model, and further reduces the parameter amount by increasing and reducing the channels of the feature map according to the Bottleneck idea.

[0049] Specifically, the three Ghost Bottlenecks modules in the backbone have the same structure, as shown in Figure 4As shown, each Ghost Bottlenecks module sequentially includes a DSC (Depthwise Separable Convolution), a Ghost module, a Depthwise Convolution, a BN, an Act, an SE Block, a BN, and the input of the Ghost Bottlenecks module is added to the output after a series of operations of the Ghost Bottlenecks module to obtain the final output of the Ghost Bottlenecks module.

[0050] wherein the DSC is to decompose a complete convolution operation into two steps, namely, a Depthwise Convolution and a Pointwise Convolution. One convolution kernel of the Depthwise Convolution is responsible for one channel, and one channel is only convolved by one convolution kernel. The Pointwise (PW) convolution is essentially a convolution with a 1x1 convolution kernel, also called a point convolution. Compared with a conventional convolution operation, the number of parameters and the operation cost are lower.

[0051] The Ghost module can generate more feature maps through inexpensive operations. On the basis of a group of feature maps, a series of linear transformations are adopted to generate many Ghost feature maps that can fully contain feature information at a lower cost. The Ghost module is divided into three parts: convolution, Ghost generation, and feature map splicing. First, a conventional 3*3 convolution is used to obtain a feature map, then a Depthwise Convolution is performed on the feature map of each channel to generate a Ghost feature map, and finally the feature map obtained in the first step and the Ghost feature map are spliced to obtain the final output. The output of the convolution layer conv contains many redundant feature maps, the Depthwise Convolution is an inexpensive operation, and the size of the feature map generated after the original convolution layer is usually small. The "ghost" of the corresponding feature map is generated through the Depthwise Convolution, and the feature map is further augmented and enhanced through the splicing operation.

[0052] The batch normalization layer BN can improve the training speed of the network and the convergence speed of the loss function. The addition of the compression activation module SE Block after the Ghost module only increases a small amount of calculation consumption, but can greatly improve the network performance, and further

[0053] The attention of the network to the channel and the segmentation accuracy of the image features of the network are improved. The core idea of the SE Block is to model the interdependence between channels, and the feature response strength between channels is adaptively corrected through the global loss function of the network, that is, the feature weight is learned by the network according to the loss, so that the effective feature map weight is large, and the weight of the ineffective or less effective feature map is small, so that the model is trained in a way to achieve better results.

[0054] The SE Block mainly includes two steps of Squeeze and Excitation, which can accurately model the dependence between channels to calibrate the convolution kernel operation.

[0055] Squeeze, that is, performing a Pooling operation on the feature map, generally a Global Average Pooling operation, to obtain global compressed feature quantities of the current feature map, and the operation aims to expand the receptive field;

[0056] Excitation, through two conv1*1 and Act operations, the weight X of each channel in the feature map is obtained, the two conv1*1 and Act operations are used to fuse the feature map information of each channel, and the weight X is the core of the SE Block, which is used to depict the weights of multiple feature maps in the feature layer, and the weight X is learned through the preceding 1*1 convolution layer and the nonlinear layer, so that it can be end-to-end trained. Each channel of the original feature layer is multiplied by the corresponding weight coefficient X to obtain the output of the SE Block, and the operation makes the proportion of effective feature maps larger and the proportion of ineffective or less effective feature maps smaller. Overall, only the values in the feature map are adjusted, and the size and the number of channels are not changed.

[0057] The SE module is a weight distribution for each channel, which is similar to the attention mechanism and helps the network to quickly learn important feature information; the module is connected to the Ghost module to help the Ghost module to fuse more features in space and extract multi-scale spatial information, so as to improve the receptive field of the Ghost module.

[0058] The MobileVit Block is added in the plurality of Ghost Bottlenecks modules, so that local representation information and global representation information in each feature map can be obtained with fewer parameters, and the expression of the features is more accurate through mutual reference between the features. In a specific embodiment, the specific structure of the MobileVit Block is as follows: Figure 5As shown in the figure, it includes DSC layer, reshape&Transpose, Transformer layer, reshape&Transpose, conv1*1 layer in sequence. The output after 1*1 convolution and the original input of MobileVit Block are concat-concatenated and then pass through the CBA layer, namely Conv layer (Convolutional layer), BN (Batch Normalization) layer and Act layer (Activation Layer), and the output of MobileVit Block is obtained.

[0059] MobileViT uses convolution and Transformer. Transformer replaces local modeling in convolution with global modeling, so that the MobileViT block has both the properties of convolution and global modeling. This makes the design of MobileViT lightweight and can achieve better performance with fewer channels and shallower networks.

[0060] Furthermore, if Figure 5 As shown in the figure, the Transformer layer consists of the following parts: the first LN (LayerNormalization) layer, the Multi-Head Attention layer, the second LN layer, the first FC (fully connected) layer, the first Act layer, the first Dropout layer, the second FC (fully connected) layer, and the second Dropout layer. The output after the Multi-Head Attention layer is residually added to the input before the first LN layer and then input into the second LN layer. The output after the second Dropout layer is residually added to the input before the second LN layer to obtain the final output.

[0061] The Transformer layer uses the self-attention mechanism to process the linear embedding sequence, extract the feature map of the entire image, and further improve the model's ability to perceive the overall features of the image. The idea of ​​the self-attention mechanism is to assign weights to each element in the feature map, and obtain the weight value of each feature through multiplication operations; the self-attention mechanism is used to obtain the vector representation of the feature, and the input vector is decoded at the feature level to highlight the importance of each part of the feature to the whole. The present invention selects a multi-head self-attention mechanism to extract image features, such as Figure 5As shown, the self-attention mechanism can be summarized as two processes in general: the first process is to calculate the weight coefficient according to Query and Key, and the second process is to weight and sum Value according to the weight coefficient. The first process can be further divided into two stages: the first stage calculates the similarity or correlation of Query and Key; the second stage introduces SoftMax to convert the score of the first stage into a numerical value, which can normalize the original calculation score to a probability distribution with the sum of all element weights being 1, and can also highlight the weight of important elements through the internal mechanism of SoftMax. After the weighted summation of Value, the final output is obtained through Transpose, reshape operation, FC layer and Dropout layer in turn. The multi-head attention mechanism is to perform multiple self-attention processing on the original input sequence, and then splice the results of each self-attention to obtain the final output result through linear transformation. The multi-head attention mechanism is an evolved version of the single-head attention mechanism, which can extract feature information from multiple dimensions to enhance the expression ability of the model.

[0062] The feature sharing structure adopted by the output head module of the application promotes the interaction between sub-tasks with relevance. In a specific embodiment, the network structure of the output head is as shown in Figure 6 As shown, the whole includes a feature sharing module and a specific task module. The feature sharing module mainly includes the aforementioned backbone network and DSC operation, and can extract features of different levels of the image through the feature sharing module. The specific task module includes a plurality of task blocks, each of which mainly includes DSC operation and a conv1*1 layer, wherein the input of the conv1*1 layer is the result of adding the features before and after the DSC operation in the task block; the results of the convolution of the plurality of task blocks are spliced to form the final output of the output head module, i.e. the avatar code.

[0063] Further, different task branches are designed in the output head module and are subdivided according to categories, and the sharing feature mechanism is also used to utilize the correlation between tasks to assist target task learning. In order to avoid the dominant influence of one or more tasks on network weights, the joint learning of all tasks should be balanced during the training process. The loss function L total is defined as follows:

[0064]

[0065] where ω i is the weight of each task, L i is a specific loss function. Random gradient descent is used to minimize the above objective, and the network weights W in the shared layersh Update to:

[0066]

[0067] From the above expression, we can see that the optimization of W sh is affected by all losses, and the influence of different losses on the shared parameters can be adjusted using the weight ω i .

[0068] When the task gradients conflict, or when the gradient amplitude of one task is much higher than that of the other tasks, the network weight update may not be optimal. This requires setting weights for different tasks in the loss, so as to ensure the relative balance of gradient amplitudes among tasks. In actual training, it is often uncertain whether the influence between each sub-task is effective, in order to eliminate the competition between multiple tasks, the gradient can be adjusted to make the training rate of each task the same, so as to automatically balance the weights in the multi-task loss function. In the specific training process, the task feature weight of the task in the step is set as:

[0069]

[0070] Where ω i is the weight of each task, N is the number of tasks, T is a constant, the larger T is, the more uniform the loss weight of each task is, and r n (t-1) is the loss at two consecutive time points t-1 and t-2, i.e. L n , the smaller the value is, the faster the training speed is. From the above expression, we can see that the weight ω i of a specific task is completely based on the change rate of the specific task loss, the faster the training speed is, the smaller the corresponding weight will be, which achieves the purpose of balancing the learning rate of each task. But such a strategy needs to balance the overall loss size in advance, otherwise some tasks may still dominate other tasks during training, so balancing the training rate and gradient size through a target can avoid this problem.

[0071] S5: As shown in FIG. 5, a customized virtual image is generated according to the virtual image code. Figure 7

[0072] ​The face kneading reasoning model in step 4 can be designed into a high-precision model through repeated attempts and optimization, but the virtual image generated directly from the coding result may have a face shape that does not match the facial features or hairstyle. In order to ensure that the customized virtual image has a higher appearance value, further processing is needed. On the one hand, the rendering effect of the virtual image is continuously debugged and optimized, such as adjusting the skin color, rendering the shadow, rendering the hair, and adjusting the highlight effect; on the other hand, various mapping materials can be designed for the virtual image, and a complete material library is constructed, such as eyebrow type, eye shape, eyelash style, face shape, and skin color. Under the collision of technical innovation and aesthetic review, the final virtual image of the method not only has high precision, but also provides a very high degree of freedom in face kneading, greatly meeting the needs of users.

[0073] In particular, the disclosed embodiments also provide a novel virtual character face kneading processing system, comprising: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following steps: obtaining a face picture of a person to be generated as the virtual character; pre-processing the face picture to obtain a model input picture through face detection and face alignment; inputting the model input picture into a gender classification model for gender recognition; selecting a corresponding face kneading reasoning module according to the gender recognition result, and then obtaining a virtual image code; and generating a customized virtual image according to the virtual image code.

[0074] In particular, the disclosed embodiments also provide a computer-readable storage medium comprising a stored computer program, wherein the program controls the device where the computer-readable storage medium is located to perform the above-mentioned virtual character face kneading method when the program is running.

[0075] In particular, the disclosed embodiments further provide a processing terminal including a storage medium and a processor configured to execute a computer program stored in the storage medium, wherein the program performs the above-mentioned virtual character face slimming method when executed. The processing terminal device in the disclosed embodiments can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The processing terminal can include a processing device (e.g., a central processing unit, a graphic processing unit, and the like) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the processing terminal are also stored. The processing device, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus. Generally, the following devices can be connected to the I / O interface: an input device including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, and the like; an output device including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; a storage device including, for example, a magnetic tape, a hard disk, and the like; and a communication device. The communication device can allow the processing terminal to perform wireless or wired communication with other devices to exchange data.

[0076] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0077] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0078] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0080] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0081] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between clients and servers arises by interplay between programs running on the respective computers and having a client-server relationship. The servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and virtual private server (VPS) services; or can be servers of a distributed system, or servers combined with a blockchain.

[0082] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by the present disclosure can be achieved, and the present disclosure is not limited herein.

[0083] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A novel virtual character face pinching method, characterized in that, The method comprises: obtaining a face picture of a person to be generated as the virtual role; preprocessing the face picture to obtain a model input picture through face detection and face alignment; inputting the model input picture into a gender classification model for gender recognition; The corresponding face pinching reasoning module is selected through the gender recognition result, and then a virtual image code is obtained, the face pinching reasoning module is divided into two types of male and female, and the network model mainly includes a backbone network for extracting image features and an output head for outputting corresponding virtual image codes; the backbone network uses a series of lightweight network modules, which sequentially include a CBA 3*3 module, a first Ghost Bottlenecks module, a MobileVit Block, a second Ghost Bottlenecks module and a third Ghost Bottlenecks module; the output head adopts a feature sharing structure, and outputs a loss function L total is: ， When optimizing, set ω i to: where ω i is the weight of the i-th task, L i is the loss function of the i-th task, N is the number of tasks, T is a constant, the larger T is, the more uniform the loss weight of each task is, r n (t-1) is the loss at two consecutive time points t-1 and t-2, that is, L n is divided by the result, and the learning rate of each task is realized by adjusting the weight ω i of each task. generating a customized virtual image according to a virtual image code.

2. The method according to claim 1, wherein, The CBA module is a convolutional network module, comprising a serially connected conv layer, a BN layer and an activation layer Act; each Ghost Bottlenecks module comprises, in sequence, a depth separable convolution DSC, a Ghost module, a depthwise convolution, a BN layer, an Act layer, an SEBlock, a BN layer, and the final output of the Ghost Bottlenecks module is obtained by adding the input of the Ghost Bottlenecks module and the output after the operation of the Ghost Bottlenecks module.

3. The method according to claim 1, wherein, The MobileVitBlock comprises, in sequence, a DSC layer, a reshape & Transpose, a Transformer layer, a reshape & Transpose, and a conv1*1 layer, and the output after the 1*1 convolution is concatenated with the original input of the MobileVit Block, and then the output of the CBA layer is obtained.

4. The method according to claim 3, wherein, The Transformer layer comprises the following parts: a first LN layer, a multi-head attention layer, a second LN layer, a first FC layer, a first Act layer, a first Dropout layer, a second FC layer, and a second Dropout layer, wherein the output after the multi-head attention layer is added to the input before the first LN layer in a residual manner, and then the output after the second Dropout layer is added to the input before the second LN layer in a residual manner to obtain the final output.

5. A novel virtual character face-kneading processing system, characterized in that, It comprises: a processor and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining a face picture of a person to be generated as the virtual role; preprocessing the face picture to obtain a model input picture through face detection and face alignment; inputting the model input picture into a gender classification model for gender recognition; The corresponding face pinching reasoning module is selected through the gender recognition result, and then a virtual image code is obtained, the face pinching reasoning module is divided into two types of male and female, and the network model mainly includes a backbone network for extracting image features and an output head for outputting corresponding virtual image codes; the backbone network uses a series of lightweight network modules, which sequentially include a CBA 3*3 module, a first Ghost Bottlenecks module, a MobileVit Block, a second Ghost Bottlenecks module and a third Ghost Bottlenecks module; the output head adopts a feature sharing structure, and outputs a loss function L total is: ， When optimizing, set ω i to: wherein ω i is the weight of the i-th task, L i is the loss function of the i-th task, N is the number of tasks, T is a constant, the larger T is, the more uniform the loss weight of each task is, r n (t-1) is the loss at two consecutive time points t-1 and t-2, i.e., L n is divided by T, and the result is obtained by adjusting the weight ω i of each task to achieve similar learning rates of each task. generating a customized virtual image according to a virtual image code.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored computer program, wherein the program controls the device where the computer-readable storage medium is located to execute the novel virtual role face shaping method of any one of claims 1 to 4 when the program is running.

7. A processing terminal, characterized by, It comprises: a storage medium and a processor, the processor being used to run a computer program stored in the storage medium, wherein the program executes the novel virtual role face shaping method of any one of claims 1 to 4 when the program is running.

Images