Cartoon digital human image generation method and device, electronic equipment and medium
By using facial vector editing and background feature replacement, the problems of character edge deformation and poor lip and eye shape driving effects in cartoon digital human character generation were solved, realizing personalized and high-fidelity digital human character generation and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CORP LTD
- Filing Date
- 2022-12-28
- Publication Date
- 2026-05-19
AI Technical Summary
Existing cartoon digital human image generation technologies suffer from poor edge deformation and lip/eye shape driving effects, limiting their usability.
By acquiring user image data, performing face detection and background separation, calculating facial parameters and background features, and using a trained image adjustment model to perform facial vector editing and background replacement, a cartoon digital human image photo is generated.
It enables the generation of personalized, high-fidelity, and diverse digital human images, lowering the barrier to entry for users and enhancing the service experience for operator customers.
Smart Images
Figure CN115880400B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for generating cartoon digital human images. Background Technology
[0002] The digital human industry is developing rapidly. In recent years, the production and operation technology of celebrity digital humans of well-known IPs has become relatively mature, and providing customized and personalized digital humans for everyone has gradually become a hot research topic in the industry. Leading Internet companies have developed automatic generation technology for personalized digital humans. Telecom operators have a broad user base and a natural application scenario demand in the field of personalized digital humans. Existing scenarios such as voice customer service, voicemail, voice prompts, and video promotions can all be upgraded using personalized digital humans.
[0003] Existing personalized digital human image generation technologies fall into three categories: one type involves allowing users to adjust parameters or match photos to generate a digital human based on a preset 3D model, resulting in low personalization; another type extracts 3D information from photos to train 3D facial meshes and skinning, resulting in low realism; another type requires scanning and recording high-quality video to create a high-precision model of a real person, which is costly and inefficient; and a third type involves uploading photos and creating digital humans based on AI-driven facial expressions. However, practical research has shown that this method has many limitations regarding the shape of the lips, eyes, and clarity of the photos, and the edges of the person and background are prone to distortion, leading to numerous usage limitations.
[0004] Therefore, the existing methods for generating cartoon digital human figures suffer from problems such as edge distortion and poor lip and eye shape driving effects, which have become urgent technical issues to be addressed.
[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0006] This disclosure provides a method, apparatus, electronic device, and storage medium for generating cartoon digital human images, which at least to some extent overcomes the problems of character edge deformation and poor lip and eye shape driving effects in the generation of cartoon digital human images in related technologies.
[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.
[0008] According to one aspect of this disclosure, a method for generating a cartoon digital human image is provided, comprising: acquiring image data uploaded by a user; performing face detection, recognition, cropping, and background separation processing on the image data to obtain a first person photo and a first background photo; determining a first triplet and a first quintuple based on the first person photo and the first background photo, wherein the first triplet is composed of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo, and the first quintuple is composed of the normalized gender and age of the person in the first person photo and the RGB mean value of the first background photo; inputting the first triplet into a trained image adjustment model to output a first cartoon image for driving the cartoon digital human, wherein the trained image adjustment model is used to perform facial vector editing on the triplet of the input image data to output the cartoon image; selecting a second background photo similar to the first background photo from a preset background library based on the first quintuple; and generating an image photo of the cartoon digital human based on the second background photo and the first cartoon image.
[0009] In one embodiment of this disclosure, determining a first triplet and a first quintet based on the first person photo and the first background photo includes: calculating facial parameters of the person in the first person photo to obtain the first triplet, wherein the facial parameters include lip opening degree, left eye opening degree, and right eye opening degree; performing AI recognition on the first person photo to determine the gender and age information of the person in the first person photo; calculating the RGB mean of the first background photo; and normalizing the gender information, age information, and the RGB mean of the first background photo to obtain the first quintet.
[0010] In one embodiment of this disclosure, the calculation of facial parameters of the person in the first person photo to obtain the first triplet includes: obtaining the coordinates of the bottom midpoint of the upper lip, the top midpoint of the lower lip, the leftmost point of the lip, and the rightmost point of the lip in the first person photo; and calculating the first lip opening degree based on the coordinates of the bottom midpoint of the upper lip, the top midpoint of the lower lip, the leftmost point of the lip, and the rightmost point of the lip.
[0011] In one embodiment of this disclosure, the preset background library contains multiple background photos and a quintuple corresponding to each background photo; selecting a second background photo similar to the first background photo from the preset background library based on the first quintuple includes: calculating the cosine similarity between the first quintuple and the quintuples of each background photo in the preset background library to obtain multiple cosine similarities; and determining the second background photo corresponding to the cosine similarity with the largest value from the multiple cosine similarities based on the magnitude of the cosine similarity values.
[0012] In one embodiment of this disclosure, the image adjustment model is trained through the following steps to obtain the trained image adjustment model: acquiring a face dataset, wherein the face dataset contains multiple face sample data; initializing the model parameters of the image adjustment model to a first parameter; setting a first threshold and a second threshold for lip opening and closing, and a third threshold and a fourth threshold for eye opening and closing; selecting a second face sample data from the face dataset as the sample data for training the image adjustment model in this round, and then performing an iterative training step; the iterative training step includes: calculating the facial parameters of the second face sample data to obtain a second triplet; performing face encoding on the second face sample data to obtain a second face vector; and determining the first parameter, the second triplet, and the second face vector to determine the second face vector. The second face vector is adjusted based on the second face vector adjustment value to obtain the adjusted second face vector; face decoding is performed on the adjusted second face vector to obtain the second cartoon image; facial parameters of the second cartoon image are calculated to obtain the second target triplet of the second cartoon image; the second residual of the second target triplet is calculated using a piecewise function, a first threshold, a second threshold, a third threshold, and a fourth threshold; the second gradient of the image adjustment model is calculated based on the second residual, and one training iteration of the image adjustment model is completed using the second gradient; face sample data is selected sequentially from the face dataset, and the iterative training steps are repeated until the image adjustment model converges to obtain the trained image adjustment model.
[0013] In one embodiment of this disclosure, the second residual includes a second mouth residual, a second left eye residual, and a second right eye residual, and the second triplet is composed of the second lip opening degree, the second left eye opening degree, and the second right eye opening degree of the person in the second face sample data.
[0014] In one embodiment of this disclosure, generating an image photo of the cartoon digital human based on the second background photo and the first cartoon image includes: performing a character-background composite on the second background photo and the first cartoon image to obtain the image photo of the cartoon digital human.
[0015] According to another aspect of this disclosure, a cartoon digital human image generation device is provided, comprising: an image data acquisition module for acquiring image data uploaded by a user; a person-background separation module for performing face detection, recognition, cropping, and background separation processing on the image data to obtain a first person photo and a first background photo; and a calculation module for determining a first triplet and a first quintuple based on the first person photo and the first background photo, wherein the first triplet is composed of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo, and the first quintuple is composed of the normalized first person photo... The system comprises the character's gender, age, and the RGB mean value of the first background photo; a cartoon image output module, used to input the first triplet into a pre-trained image adjustment model and output a first cartoon image for driving the cartoon digital human, wherein the pre-trained image adjustment model is used to perform facial vector editing on the triplet of the input image data to output the cartoon image; a similar background selection module, used to select a second background photo similar to the first background photo from a preset background library based on the first quintuple; and an image photo generation module, used to generate an image photo of the cartoon digital human based on the second background photo and the first cartoon image.
[0016] In one embodiment of this disclosure, the above-mentioned calculation module includes a facial parameter calculation module and a background feature calculation module. The facial parameter calculation module is used to calculate the facial parameters of the person in the first person photo to obtain the first triplet. The facial parameters include lip opening degree, left eye opening degree, and right eye opening degree. The background feature calculation module is used to perform AI recognition on the first person photo to determine the gender and age information of the person in the first person photo, calculate the RGB mean of the first background photo, and normalize the gender information, age information, and the RGB mean of the first background photo to obtain the first quintuple.
[0017] In one embodiment of this disclosure, the facial parameter calculation module is further configured to obtain the coordinates of the bottom midpoint of the upper lip, the top midpoint of the lower lip, the leftmost point of the lip, and the rightmost point of the lip in the first person photo; and to calculate the first lip opening degree based on the coordinates of the bottom midpoint of the upper lip, the top midpoint of the lower lip, the leftmost point of the lip, and the rightmost point of the lip.
[0018] In one embodiment of this disclosure, the preset background library contains multiple background photos and a quintuple corresponding to each background photo; the similar background selection module is further configured to calculate the cosine similarity between the first quintuple and the quintuples of each background photo in the preset background library to obtain multiple cosine similarities; and determine the second background photo corresponding to the largest cosine similarity from the multiple cosine similarities based on the magnitude of the cosine similarity values.
[0019] In one embodiment of this disclosure, the above-mentioned apparatus further includes a model training module, which is used to train an image adjustment model through the following steps to obtain the trained image adjustment model: acquiring a face dataset, wherein the face dataset contains multiple face sample data; initializing the model parameters of the image adjustment model to a first parameter; setting a first threshold and a second threshold for lip opening and closing, and a third threshold and a fourth threshold for eye opening and closing; selecting a second face sample data from the face dataset as the sample data for training the image adjustment model in this round, and then performing an iterative training step; the iterative training step includes: calculating the facial parameters of the second face sample data to obtain a second triplet; performing face encoding on the second face sample data to obtain a second face vector; and according to the first parameter, the second triplet, and... The second face vector is used to determine the adjustment value of the second face vector; the second face vector of the second face sample data is adjusted according to the adjustment value of the second face vector to obtain the adjusted second face vector; face decoding is performed on the adjusted second face vector to obtain the second cartoon image; the facial parameters of the second cartoon image are calculated to obtain the second target triplet of the second cartoon image; the second residual of the second target triplet is calculated using a piecewise function, a first threshold, a second threshold, a third threshold, and a fourth threshold; the second gradient of the image adjustment model is calculated based on the second residual, and the second gradient is used to complete one training iteration of the image adjustment model; face sample data is selected sequentially from the face dataset, and the iterative training steps are repeated until the image adjustment model converges to obtain the trained image adjustment model.
[0020] In one embodiment of this disclosure, the above-mentioned image photo generation module is further used to perform character and background compositing on the second background photo and the first cartoon image to obtain the image photo of the cartoon digital human.
[0021] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described method for generating a cartoon digital human image by executing the executable instructions.
[0022] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method for generating cartoon digital human images.
[0023] This disclosure provides a method, apparatus, electronic device, and storage medium for generating cartoon digital human images. The method includes: acquiring image data uploaded by a user; performing face detection, recognition, cropping, and background separation processing on the image data to obtain a first character photo and a first background photo; determining a first triplet and a first quintuple based on the first character photo and the first background photo; inputting the first triplet into a trained image adjustment model to output a first cartoon image for driving the cartoon digital human; selecting a second background photo similar to the first background photo from a preset background library based on the first quintuple; and generating a cartoon digital human image photo based on the second background photo and the first cartoon image. This disclosure solves the problems of edge distortion, poor lip and eye shape driving effects, and flawed edge and background driving in existing cartoon digital human image generation methods.
[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0026] Figure 1 This diagram illustrates a communication system architecture according to an embodiment of the present disclosure;
[0027] Figure 2 This diagram illustrates a flowchart of a method for generating a cartoon digital human image according to an embodiment of the present disclosure.
[0028] Figure 3 This diagram illustrates a method for generating a cartoon digital human image according to an embodiment of the present disclosure.
[0029] Figure 4 This diagram illustrates another method for generating a cartoon digital human image in an embodiment of this disclosure.
[0030] Figure 5 This diagram illustrates another method for generating a cartoon digital human image in an embodiment of this disclosure.
[0031] Figure 6This diagram illustrates a cartoon digital human image generation device according to an embodiment of the present disclosure.
[0032] Figure 7 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation
[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0034] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0035] As mentioned in the background section regarding a method, device, electronic device, and storage medium for generating cartoon digital human images, existing personalized digital human image generation technologies fall into three categories: one type involves allowing users to adjust parameters or match photos to generate digital humans based on a preset 3D model, resulting in low personalization; another type involves extracting 3D information from photos to train 3D facial meshes and skinning, resulting in low realism; another type requires scanning and recording high-quality videos to create high-precision models of real people, which is costly and inefficient; and yet another type involves uploading photos and creating digital humans based on AI-driven facial expressions. However, practical research has shown that this type of method has many limitations regarding the shape of lips, eyes, and clarity of photos, and the edges of the person and background are prone to distortion, leading to numerous usage limitations.
[0036] Based on this, this disclosure provides a method, apparatus, electronic device, and storage medium for generating cartoon digital human images. Addressing the personalized digital human image generation scenarios of operators, and their needs for personalization, low cost, and low barriers to entry, this disclosure innovatively proposes a face editing method and a background feature extraction and replacement method based on vector editing prediction, building upon existing comic-style encoding and decoding technologies. This solves the technical limitations of digital human video-driven technology, such as edge distortion and poor lip and eye shape driving effects, enabling personalized, high-fidelity, and diverse digital human image generation. This lowers the user threshold and enhances the service experience for operator customers.
[0037] This disclosure addresses the limitations of lip and eye pose in digital human expression-driven methods by incorporating facial vector editing steps into the stylized transfer of face encoding and decoding. It reduces the limitations on lip and eye poses in digital human cartoon-driven images, expanding the application scenarios of this technology. Existing single-image expression-driven methods fail to adequately address background filling at the edges when a person is moving. This disclosure, targeting digital human-driven scenarios, solves the problem of background filling defects at the edges of the image.
[0038] This disclosure proposes a background calculation method based on background and human features, enhancing the personalization of background images and overcoming the technical limitations of background distortion at the edges of human figures in digital human-driven scenarios. This disclosure also proposes a prediction model and training method for facial editing parameters, enabling precise editing through a single calculation via parameter prediction.
[0039] Figure 1 A schematic diagram of an exemplary system architecture for a cartoon digital human image generation method or apparatus that can be applied to embodiments of this disclosure is shown.
[0040] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105.
[0041] Network 104 is a medium used to provide a communication link between terminal devices 101, 102, 103 and server 105, and can be a wired network or a wireless network.
[0042] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.
[0043] Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.
[0044] Optionally, the client applications installed on different terminal devices 101, 102, and 103 may be the same, or clients of the same type of application based on different operating systems. Depending on the terminal platform, the specific form of the application client may also differ; for example, the application client may be a mobile client, a PC client, etc.
[0045] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.
[0046] Optionally, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.
[0047] Those skilled in the art will know that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; any number of terminal devices, networks, and servers can be included depending on actual needs. This disclosure does not limit the scope of the embodiments.
[0048] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.
[0049] First, this disclosure provides a method for generating a cartoon digital human image, which can be executed by any electronic device with computing power.
[0050] Figure 2 This diagram illustrates a flowchart of a method for generating a cartoon digital human image according to an embodiment of the present disclosure, such as... Figure 2 As shown, the method for generating a cartoon digital human image provided in this embodiment includes the following steps:
[0051] S202, Obtain image data uploaded by the user.
[0052] It should be noted that the image data can be photos in various image formats that contain facial images, including formats such as jpg, png, gif, psd, tif, and bmp.
[0053] S204, perform face detection, recognition, cropping, and background separation processing on the image data to obtain the first person photo and the first background photo.
[0054] It should be noted that this step can separate the person from the background in the image data to obtain a first person photo and a first background photo. The first person photo can be a photo containing only person information, and the first background photo can be a photo containing only background information.
[0055] S206. Based on the first person photo and the first background photo, determine the first triplet and the first quintet. The first triplet consists of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo. The first quintet consists of the normalized gender and age of the person in the first person photo and the RGB mean of the first background photo.
[0056] In one embodiment of this disclosure, determining the first triplet and the first quintet based on the first person photograph and the first background photograph includes:
[0057] Calculate the facial parameters of the person in the first photograph to obtain the first triplet, where the facial parameters include lip opening degree, left eye opening degree, and right eye opening degree.
[0058] AI recognition is used to identify the gender and age of the person in the first photo. The RGB mean of the first background photo is calculated, and the gender, age and RGB mean of the first background photo are normalized to obtain the first quintuple.
[0059] It should be noted that before calculating the facial parameters of the person in the first portrait photo to obtain the first triplet, the first portrait photo can be preprocessed. Preprocessing can include cropping, face alignment, and scaling to obtain a portrait photo that meets the size requirements. After preprocessing the first portrait photo, the first degree of lip opening, the first degree of left eye opening, and the first degree of right eye opening are calculated to obtain the first triplet. AI recognition is then performed on the first portrait photo to obtain the gender and normalize it to 's', and the age is normalized to 'a'. The RGB mean of the first background photo is calculated and normalized to obtain 'r', 'g', and 'b', forming the first quintuple (s, a, r, g, b).
[0060] In one embodiment of this disclosure, facial parameters of a person in a first photograph are calculated to obtain a first triplet, including:
[0061] Obtain the coordinates of the bottom center of the upper lip, the top center of the lower lip, the leftmost point of the lip, and the rightmost point of the lip in the first person's photo.
[0062] The first lip opening degree is calculated based on the coordinates of the midpoint of the bottom of the upper lip, the midpoint of the top of the lower lip, the leftmost point of the lip, and the rightmost point of the lip.
[0063] It should be noted that, see Figure 3The diagram illustrates a method for generating a cartoon digital human image. It shows how a facial key point algorithm can be used to obtain the facial key points of a person in a first-person photo. Here, facial key points can include the coordinates of the bottom center of the upper lip, the top center of the lower lip, the leftmost point of the lip, the rightmost point of the lip, the top center of the left eye, the top center of the right eye, the bottom center of the left eye, the bottom center of the right eye, the leftmost point of the left eye, the rightmost point of the right eye, and so on.
[0064] Taking the calculation of lip opening as an example, obtain the coordinates of the midpoint of the bottom of the upper lip (mt(x, y), the midpoint of the top of the lower lip (mb(x, y), the leftmost point of the lip (ml(x, y), and the rightmost point of the lip (mr(x, y)); calculate the lip tilt mc = arctan((mry - mly) / (mrx - mlx)); calculate the lip opening m = cosin(mc) × ((mty - mby) × (mty - mby) + (mtx - mbx) × () mtx-mbx); where mry is the ordinate of the rightmost point of the lip mr, mly is the ordinate of the leftmost point of the lip ml, mrx is the abscissa of the rightmost point of the lip mr, mlx is the abscissa of the leftmost point of the lip ml, mty is the ordinate of the midpoint of the bottom of the upper lip mt, mby is the ordinate of the midpoint of the top of the lower lip mb, mtx is the abscissa of the midpoint of the bottom of the upper lip mt, and mbx is the abscissa of the midpoint of the top of the lower lip mb. Similarly, based on the midpoint of the top of the left eye et1, the midpoint of the top of the right eye et2, the midpoint of the bottom of the left eye eb1, the midpoint of the bottom of the right eye eb2, the leftmost point of the left eye el1, the leftmost point of the right eye el2, the rightmost point of the left eye er1, and the rightmost point of the right eye er2, the tilt of the left eye me1 and the tilt of the right eye me2 can be calculated, and then the opening degree of the left eye e1 and the opening degree of the right eye e2 can be calculated.
[0065] S208, the first triplet is input into the trained image adjustment model, and the first cartoon image for cartoon digital human driving is output. The trained image adjustment model is used to edit the facial vectors of the triplet of the input image data to output the cartoon image.
[0066] It should be noted that the trained image adjustment model uses triples of image data as input and cartoon images as output; the first cartoon image can be a 2D digital human image photograph in a cartoon style.
[0067] S210, based on the first quintuple, select a second background photo that is similar to the first background photo from the preset background library.
[0068] In one embodiment of this disclosure, a preset background library contains multiple background photos and a quintuple corresponding to each background photo; selecting a second background photo similar to the first background photo from the preset background library based on the first quintuple includes: calculating the cosine similarity between the first quintuple and the quintuples of each background photo in the preset background library to obtain multiple cosine similarities; and determining the second background photo corresponding to the cosine similarity with the largest value from the multiple cosine similarities based on the magnitude of the cosine similarity values.
[0069] S212, Generate a cartoon digital human image photo based on the second background photo and the first cartoon image.
[0070] The cartoon digital human image generation method provided in this disclosure, based on vector editing prediction of facial editing and background feature extraction and replacement, solves the technical limitations of digital human video driving technology, such as character edge deformation and poor lip and eye shape driving effects. It realizes personalized, high-fidelity, and diverse digital human image generation, lowers the user threshold, and improves the service experience of operator customers.
[0071] In one embodiment of this disclosure, an image adjustment model is trained through the following steps to obtain a trained image adjustment model:
[0072] Obtain a face dataset, which contains multiple face sample data.
[0073] Initialize the model parameters of the image adjustment model to the first parameter.
[0074] Set a first and a second threshold for lip opening and a third and a fourth threshold for eye opening and closing.
[0075] After selecting the second face sample data from the face dataset as the sample data for training the image adjustment model in this round, the iterative training steps are executed.
[0076] The iterative training steps include: calculating the facial parameters of the second face sample data to obtain the second triplet; performing face encoding on the second face sample data to obtain the second face vector; determining the adjustment value of the second face vector based on the first parameter, the second triplet, and the second face vector; adjusting the second face vector of the second face sample data based on the second face vector adjustment value to obtain the adjusted second face vector; performing face decoding on the adjusted second face vector to obtain the second cartoon image; calculating the facial parameters of the second cartoon image to obtain the second target triplet of the second cartoon image; calculating the second residual of the second target triplet using a piecewise function, a first threshold, a second threshold, a third threshold, and a fourth threshold; calculating the second gradient of the image adjustment model based on the second residual; and completing one training iteration of the image adjustment model using the second gradient.
[0077] After selecting face sample data from the face dataset in sequence, the iterative training steps are repeated until the image adjustment model converges, and the trained image adjustment model is obtained.
[0078] It should be noted that after constructing the face dataset, it can be divided into a training set and a test set. The training set is used for training the image adjustment model, and the test set is used to test the output effect of the trained image adjustment model. The face dataset contains multiple face samples of the same size after cropping, face alignment, and scaling. The face data samples can be photos containing face images. The first threshold can be the maximum threshold for lip opening, the second threshold can be the minimum threshold for lip opening, the third threshold can be the maximum threshold for eye opening, and the fourth threshold can be the minimum threshold for eye opening.
[0079] In one embodiment of this disclosure, see Figure 4 The diagram illustrates another method for generating a cartoon digital human. The model parameters of the image adjustment model are initialized to a first parameter M. The image adjustment model can use, but is not limited to, neural network models, tree models, or linear models. A first threshold mMax and a second threshold mMin are set for lip opening, and a third threshold eMax and a fourth threshold eMin are set for eye opening. Second face sample data is selected from the face dataset as the sample data for training the image adjustment model in this round. Facial parameters of the second face sample data are calculated to obtain a second triplet. Face encoding is performed on the second face sample data to obtain a second face vector, where... Face encoding can be performed using methods such as pixel2pixel or StyleGAN. Based on the first parameter M, the second triplet F, and the second face vector V, the adjustment value dF of the second face vector is determined. Using the second face vector adjustment value dF as a parameter, a face vector editing model is used to modify the face vector V, resulting in the adjusted second face vector V'. The face vector editing model is used to adjust the face vector based on the adjustment value. A comic book decoder is used to perform face decoding on the adjusted second face vector V' to obtain the second comic book image R. The facial parameters of the second comic book image R are calculated to obtain the second target triplet F'.
[0080] Using a piecewise function, a first threshold, a second threshold, a third threshold, and a fourth threshold, the second residual C of the second target triplet F' is calculated. The second gradient is then calculated based on the second residual C. One training iteration of the image adjustment model is completed using the second gradient. Face sample data is selected sequentially from the face dataset, and the iterative training steps are repeated until the image adjustment model converges, resulting in a trained image adjustment model.
[0081] In one embodiment of the present disclosure, the second residual includes a second mouth residual, a second left-eye residual, and a second right-eye residual, and the second triple is composed of the second lip opening degree, the second left-eye opening degree, and the second right-eye opening degree of the person in the second face sample data.
[0082] It should be noted that a piecewise function is used to calculate the second residual C for the second target triple F’(m, e1, e2). When m < mMin, the second mouth residual cm = mMin - F’m; when m > mMin, cm = F’m - mMax; when m is less than or equal to mMax and greater than or equal to mMin, cm = 0. Similarly, the second left-eye residual ce1 and the second right-eye residual ce2 can be calculated. According to the second mouth residual, the second left-eye residual, and the second right-eye residual, the second residual is calculated, and the second residual C = sqrt(ce1×ce1 + ce2×ce2 + cm×cm).
[0083] In one embodiment of the present disclosure, according to the second background photo and the first comic image, a profile photo of a cartoon digital human is generated, including: performing person-background synthesis on the second background photo and the first comic image to obtain the profile photo of the cartoon digital human.
[0084] In one embodiment of the present disclosure, the digital human platform app prompts the user to upload a photo; the user selects a photo from the mobile phone album and uploads it; the digital human platform uses the method of the present disclosure to generate profile photos of digital humans in various comic styles for the user to choose from; the user selects a favorite comic image as their digital human image; the digital human platform uses the selected digital human image to generate multiple segments of digital human speaking videos for digital human playback and message guidance when the user does not answer the phone. When the user does not answer an incoming call, the digital human dialogue module of the digital human platform takes over the calling signaling, plays the pre-generated multi-round digital human dialogue videos, and guides the user to leave a voice message. When user A sees an incoming call in the app, the platform obtains the audio and text message information of user B stored previously.
[0085] In one embodiment of the present disclosure, refer to Figure 5 As shown in the schematic diagram of another method for generating the profile of a cartoon digital human, after face detection, cropping, and background separation of the photo, a person photo and a background photo can be obtained. The facial parameters of the person in the person photo can be calculated to obtain a triple, and the person photo is face-encoded to obtain a face vector; based on the image adjustment model and the triple, a predicted facial editing vector is determined, the face vector is adjusted according to the predicted facial editing vector to obtain an adjusted face vector, and the adjusted face vector is face-decoded to obtain a comic picture; based on the background photo, a similar background photo is selected from a preset background library to obtain a similar background photo; the comic picture and the similar background photo are synthesized to obtain a profile photo for driving the cartoon digital human.
[0086] Based on the same inventive concept, this disclosure also provides a cartoon digital human image generation device, as shown in the following embodiment. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiment, the implementation of this device embodiment can refer to the implementation of the above-described method embodiment, and repeated details will not be elaborated further.
[0087] Figure 6 This diagram illustrates a cartoon digital human image generation device according to an embodiment of the present disclosure, such as... Figure 6 As shown, the device includes:
[0088] Image data acquisition module 610 is used to acquire image data uploaded by the user;
[0089] The person and background separation module 620 is used to perform face detection, recognition, cropping and background separation processing on image data to obtain a first person photo and a first background photo;
[0090] The calculation module 630 is used to determine a first triplet and a first quintet based on the first person photo and the first background photo. The first triplet consists of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo. The first quintet consists of the normalized gender and age of the person in the first person photo and the RGB mean value of the first background photo.
[0091] The cartoon image output module 640 is used to input the first triplet into the trained image adjustment model and output the first cartoon image for cartoon digital human driving. The trained image adjustment model is used to perform facial vector editing on the triplet of the input image data to output the cartoon image.
[0092] The similar background selection module 650 is used to select a second background photo that is similar to the first background photo from a preset background library based on the first quintuple;
[0093] Image photo generation module 660 is used to generate image photos of cartoon digital humans based on a second background photo and a first cartoon image.
[0094] In one embodiment of this disclosure, the above-mentioned calculation module 630 includes a facial parameter calculation module and a background feature calculation module. The facial parameter calculation module is used to calculate the facial parameters of the person in the first person photo to obtain a first triplet, wherein the facial parameters include the degree of lip opening and closing, the degree of left eye opening and closing, and the degree of right eye opening and closing. The background feature calculation module is used to perform AI recognition on the first person photo to determine the gender information and age information of the person in the first person photo, calculate the RGB mean of the first background photo, and normalize the gender information, age information, and the RGB mean of the first background photo to obtain a first quintuple.
[0095] In one embodiment of this disclosure, the facial parameter calculation module is further used to obtain the coordinates of the bottom midpoint of the upper lip, the top midpoint of the lower lip, the leftmost point of the lip, and the rightmost point of the lip in the first person photo; and to calculate the first lip opening degree based on the coordinates of the bottom midpoint of the upper lip, the top midpoint of the lower lip, the leftmost point of the lip, and the rightmost point of the lip.
[0096] In one embodiment of this disclosure, the preset background library contains multiple background photos and a quintuple corresponding to each background photo; the aforementioned similar background selection module 650 is further configured to calculate the cosine similarity between the first quintuple and the quintuples of each background photo in the preset background library, thereby obtaining multiple cosine similarities; and based on the magnitude of the cosine similarity values, determine the second background photo corresponding to the cosine similarity with the largest value from among the multiple cosine similarities.
[0097] In one embodiment of this disclosure, the apparatus further includes a model training module, which is used to train an image adjustment model through the following steps to obtain a trained image adjustment model: acquiring a face dataset, wherein the face dataset contains multiple face sample data; initializing the model parameters of the image adjustment model to a first parameter; setting a first threshold and a second threshold for lip opening and closing, and a third threshold and a fourth threshold for eye opening and closing; selecting second face sample data from the face dataset as sample data for training the image adjustment model in this round, and then performing an iterative training step; the iterative training step includes: calculating the facial parameters of the second face sample data to obtain a second triplet; performing face encoding on the second face sample data to obtain a second face vector; and according to the first parameter, The second triplet and the second face vector are used to determine the adjustment value of the second face vector. The second face vector of the second face sample data is adjusted according to the second face vector adjustment value to obtain the adjusted second face vector. Face decoding is performed on the adjusted second face vector to obtain the second cartoon image. The facial parameters of the second cartoon image are calculated to obtain the second target triplet of the second cartoon image. The second residual of the second target triplet is calculated using a piecewise function, a first threshold, a second threshold, a third threshold, and a fourth threshold. The second gradient of the image adjustment model is calculated based on the second residual, and one training iteration of the image adjustment model is completed using the second gradient. Face sample data is selected sequentially from the face dataset, and the iterative training steps are repeated until the image adjustment model converges, resulting in a trained image adjustment model.
[0098] In one embodiment of this disclosure, the image photo generation module 660 is further used to perform character and background synthesis on the second background photo and the first cartoon image to obtain an image photo of a cartoon digital person.
[0099] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."
[0100] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.
[0101] like Figure 7As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).
[0102] The storage unit stores program code, which can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure. For example, the processing unit 710 can execute the following steps of the above method embodiment: acquiring image data uploaded by the user; performing face detection, recognition, cropping, and background separation processing on the image data to obtain a first person photo and a first background photo; determining a first triplet and a first quintuple based on the first person photo and the first background photo, wherein the first triplet consists of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo, and the first quintuple consists of the normalized gender and age of the person in the first person photo and the RGB mean value of the first background photo; inputting the first triplet into a trained image adjustment model to output a first cartoon image for driving a cartoon digital human, wherein the trained image adjustment model is used to perform facial vector editing on the triplet of the input image data to output a cartoon image; selecting a second background photo similar to the first background photo from a preset background library based on the first quintuple; and generating an image photo of the cartoon digital human based on the second background photo and the first cartoon image.
[0103] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.
[0104] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0105] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0106] Electronic device 700 can also communicate with one or more external devices 740 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0107] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0108] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.
[0109] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0110] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.
[0111] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0112] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0113] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0114] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0115] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0116] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A method for generating a cartoon digital human image, characterized in that, include: Get image data uploaded by the user; The image data is subjected to face detection, recognition, cropping, and background separation processing to obtain a first person photo and a first background photo; Based on the first person photo and the first background photo, a first triplet and a first quintet are determined. The first triplet consists of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo. The first quintet consists of the normalized gender and age of the person in the first person photo and the RGB mean value of the first background photo. The first triplet is input into the trained image adjustment model, and the output is a first cartoon image for cartoon digital human driving. The trained image adjustment model is used to perform facial vector editing on the triplet of the input image data to output the cartoon image. Based on the first quintuple, select a second background photo that is similar to the first background photo from the preset background library; A photograph of the cartoon digital human is generated based on the second background photograph and the first cartoon image.
2. The method for generating cartoon digital human images according to claim 1, characterized in that, Based on the first person photo and the first background photo, determine the first triplet and the first quintet, including: Calculate the facial parameters of the person in the first person photo to obtain the first triplet, wherein the facial parameters include lip opening degree, left eye opening degree and right eye opening degree; AI recognition is performed on the first person photo to determine the gender and age information of the person in the first person photo. The RGB mean of the first background photo is calculated, and the gender information, age information and the RGB mean of the first background photo are normalized to obtain the first quintuple.
3. The method for generating cartoon digital human images according to claim 2, characterized in that, Calculate the facial parameters of the person in the first photograph to obtain the first triplet, which includes: Obtain the coordinates of the bottom center point of the upper lip, the top center point of the lower lip, the leftmost point of the lip, and the rightmost point of the lip in the first person's photo; The first lip opening degree is calculated based on the coordinates of the midpoint of the bottom of the upper lip, the midpoint of the top of the lower lip, the leftmost point of the lip, and the rightmost point of the lip.
4. The method for generating a cartoon digital human image according to claim 1, characterized in that, The preset background library contains multiple background photos and a quintuple corresponding to each background photo; Based on the first quintuple, a second background photo similar to the first background photo is selected from a preset background library, including: Calculate the cosine similarity between the first quintuple and the quintuples of each background photo in the preset background library to obtain multiple cosine similarities; Based on the magnitude of the cosine similarity, the second background photo corresponding to the largest cosine similarity value is determined from the plurality of cosine similarities.
5. The method for generating a cartoon digital human image according to claim 1, characterized in that, The trained image adjustment model is obtained by training the image adjustment model through the following steps: Obtain a face dataset, wherein the face dataset contains multiple face sample data; Initialize the model parameters of the image adjustment model to the first parameter; Set a first threshold and a second threshold for lip opening and closing, and a third threshold and a fourth threshold for eye opening and closing; After selecting the second face sample data from the face dataset as the sample data for training the image adjustment model in this round, the iterative training step is executed; The iterative training steps include: calculating facial parameters of the second face sample data to obtain a second triplet; performing face encoding on the second face sample data to obtain a second face vector; determining a second face vector adjustment value based on the first parameter, the second triplet, and the second face vector; adjusting the second face vector of the second face sample data based on the second face vector adjustment value to obtain an adjusted second face vector; performing face decoding on the adjusted second face vector to obtain a second cartoon image; calculating facial parameters of the second cartoon image to obtain a second target triplet of the second cartoon image; calculating a second residual of the second target triplet using a piecewise function, a first threshold, a second threshold, a third threshold, and a fourth threshold; calculating a second gradient of the image adjustment model based on the second residual; and completing one training iteration of the image adjustment model using the second gradient. After selecting face sample data from the face dataset in sequence, the iterative training steps are repeated until the image adjustment model converges, thus obtaining the trained image adjustment model.
6. The method for generating a cartoon digital human image according to claim 5, characterized in that, The second residual includes the second mouth residual, the second left eye residual, and the second right eye residual. The second triplet is composed of the second lip opening degree, the second left eye opening degree, and the second right eye opening degree of the person in the second face sample data.
7. The method for generating a cartoon digital human image according to claim 1, characterized in that, Based on the second background photo and the first cartoon image, generate an image photo of the cartoon digital human, including: The second background photo and the first cartoon image are combined to obtain the image photo of the cartoon digital human.
8. A device for generating the image of a cartoon digital human, characterized in that, include: The image data acquisition module is used to acquire image data uploaded by the user; The person and background separation module is used to perform face detection, recognition, cropping and background separation processing on the image data to obtain a first person photo and a first background photo; The calculation module is used to determine a first triplet and a first quintet based on the first person photo and the first background photo. The first triplet consists of the first lip opening degree, the first left eye opening degree, and the first right eye opening degree of the person in the first person photo. The first quintet consists of the normalized gender and age of the person in the first person photo and the RGB mean value of the first background photo. A cartoon image output module is used to input the first triplet into a trained image adjustment model and output a first cartoon image for cartoon digital human driving, wherein the trained image adjustment model is used to perform facial vector editing on the triplet of the input image data to output a cartoon image; The similar background selection module is used to select a second background photo that is similar to the first background photo from a preset background library based on the first quintuple; The image photo generation module is used to generate an image photo of the cartoon digital human based on the second background photo and the first cartoon image.
9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the cartoon digital human image generation method according to any one of claims 1 to 7 by executing the executable instructions.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for generating cartoon digital human images according to any one of claims 1 to 7.