Digital human customization method, cloud end, vehicle machine end, equipment, medium and product

Abstract

The invention provides a digital human customization method, a cloud end, a vehicle machine end, equipment, a medium and a product. According to one example of the invention, the method is applied to a cloud, and the method can comprise the following steps: recognizing a portrait picture uploaded by a user, and determining an image template matched with the portrait picture; performing feature fusion on the portrait picture and the image template to generate a digital human picture; carrying out portrait replacement on a motion capture video template corresponding to the image template by using the digital human picture to generate digital human data; and the digital person data is sent to the in-vehicle terminal, so that the in-vehicle terminal displays the digital person in the digital person data, a unique digital person image can be generated and displayed in a customized manner, and the personalized experience of a user on the digital person is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a digital human customization method, cloud, vehicle terminal, equipment, medium and product. Background Art

[0002] With the rapid development of automobile electrification and intelligent technology, cars have gradually surpassed their traditional role as a means of transportation and have become a "mobile home" that combines comfort, entertainment and convenience. In this transformation process, intelligent functions are increasingly used in automobile development, significantly improving users' driving experience and riding enjoyment.

[0003] Currently, a digital human (i.e., virtual digital assistant) is usually built into the intelligent system of a car as a bridge of communication between the user and the vehicle. The digital human uses voice recognition and interaction technology to control multiple functions of the vehicle, including but not limited to: music playback, video on demand, window control, etc. This non-contact control method greatly improves the safety and convenience of driving, allowing the driver to complete a series of operations without distraction during driving.

[0004] However, the image of a digital human is often statically configured when the car leaves the factory and lacks a personalized experience. Summary of the invention

[0005] In order to overcome the problems existing in the related technologies, the present application provides a digital human customization method, cloud, vehicle terminal, equipment, medium and product.

[0006] According to a first aspect of any embodiment of the present application, a method for customizing a digital human is provided, the method being applied in a cloud, and the method comprising:

[0007] Identify the portrait picture uploaded by the user and determine the image template that matches the portrait picture;

[0008] Fusing the portrait image with the image template to generate a digital human image;

[0009] Using the digital human image, performing a human portrait replacement on the motion capture video template corresponding to the image template to generate digital human data;

[0010] The digital human data is sent to the vehicle terminal, so that the vehicle terminal displays the digital human in the digital human data.

[0011] According to a second aspect of any embodiment of the present application, a digital human customization method is provided, the method being applied to a vehicle terminal, the method comprising:

[0012] Receiving digital human data sent by the cloud; wherein the digital human data is generated by fusing features of a portrait picture uploaded by a user with a matching image template, and performing portrait replacement on a motion capture video template corresponding to the image template;

[0013] The digital human in the digital human data is displayed.

[0014] According to a third aspect of any embodiment of the present application, a cloud is provided, the cloud comprising:

[0015] A recognition module, used to recognize the portrait picture uploaded by the user and determine the image template that matches the portrait picture;

[0016] A fusion module, used to fuse the features of the portrait image with the image template to generate a digital human image;

[0017] A generating module, configured to use the digital human image to replace the motion capture video template corresponding to the image template with the human image, thereby generating digital human data;

[0018] The processing module is used to send the digital human data to the vehicle terminal so that the vehicle terminal displays the digital human in the digital human data.

[0019] According to a fourth aspect of any one of the embodiments of the present application, a vehicle terminal is provided, the vehicle terminal comprising:

[0020] A receiving module, used to receive digital human data sent from the cloud; wherein the digital human data is generated by fusing the features of the portrait image uploaded by the user with the matching image template, and replacing the portrait with the motion capture video template corresponding to the image template;

[0021] The display module is used to display the digital human in the digital human data.

[0022] According to a fifth aspect of any embodiment of the present application, there is provided an electronic device, including:

[0023] processor;

[0024] a memory for storing processor-executable instructions;

[0025] The processor runs the executable instructions to implement a method executed by the cloud or a method executed by the vehicle terminal in any of the above-mentioned embodiments of the present application.

[0026] According to the sixth aspect of any embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, a method executed by the cloud or a method executed by the vehicle terminal in any embodiment of the present application is implemented.

[0027] According to the seventh aspect of any embodiment of the present application, a computer program product is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, it implements the method executed by the cloud or the method executed by the vehicle end in any embodiment of the present application as described above.

[0028] The technical solution provided by this application may have the following beneficial effects:

[0029] According to the above embodiments, by identifying the portrait picture uploaded by the user, determining the image template that matches the portrait picture, fusing the features of the portrait picture and the image template, generating a digital human picture, using the digital human picture, replacing the portrait in the motion capture video template corresponding to the image template, generating digital human data, and sending the digital human data to the vehicle computer so that the vehicle computer displays the digital human in the digital human data. A unique digital human image can be customized and displayed, thereby improving the user's personalized experience of the digital human.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0032] Figure 1 is a flow chart of a digital human customization method according to an exemplary embodiment of the present application;

[0033] Figure 2 is a flow chart of another digital human customization method according to an exemplary embodiment of the present application;

[0034] Figure 3 is a flowchart of a stylized rendering method according to an exemplary embodiment of the present application;

[0035] Figure 4 is an interactive diagram of a digital human customization method according to an exemplary embodiment of the present application;

[0036] Figure 5 is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present application;

[0037] Figure 6 is a block diagram of a cloud according to an exemplary embodiment of the present application;

[0038] Figure 7 It is a block diagram of a vehicle terminal shown in the present application according to an exemplary embodiment. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0040] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0042] At present, the image of digital humans in automobile intelligent systems is often statically configured when the car leaves the factory and lacks a personalized experience.

[0043] In order to solve the above problems, the present application proposes a digital human customization method. To further illustrate the present application, the following embodiments are provided:

[0044] See also Figure 1 , Figure 1 This is a flow chart of a digital human customization method according to an exemplary embodiment of the present application. The digital human customization method can be applied to the cloud, and the digital human customization method may include the following steps:

[0045] Step 102: Identify the portrait picture uploaded by the user and determine an image template that matches the portrait picture.

[0046] In this step, the user can upload the portrait picture through the user end such as the application program, mini program, etc. For example, the user scans the QR code on the vehicle end through the mobile phone to log in to the mini program and upload the portrait picture with a clear face.

[0047] The cloud obtains the portrait pictures uploaded by the user, and can firstly perform pre-processing such as reviewing and cropping on the portrait pictures, and then perform recognition processing on the portrait pictures. Exemplarily, the recognition technology of the portrait pictures can use deep learning algorithms, such as Convolutional Neural Networks (CNN), to extract the portrait features in the portrait pictures, such as age, gender, face shape, hairstyle, facial features, facial features, etc.

[0048] The cloud can match the pre-stored image template library based on portrait features and determine the image template that best matches the portrait picture, so as to provide users with a digital representation that is closest to their real image, effectively improving the similarity between the digital human image and the portrait picture uploaded by the user.

[0049] The cloud can also display various image templates in the image template library through the user end or the vehicle end, and receive the image template selected by the user.

[0050] Among them, the portrait image is a photo containing the face uploaded by the user, which is used to extract the portrait features for subsequent recognition and matching. The image template is used to provide the basic image framework of the digital human. The image template can include specific image features, such as: gender, age, hairstyle, face shape, etc. The image template can have a solid color background to facilitate subsequent feature fusion.

[0051] Each image template in the image template library is an image that meets product requirements and is generated through a diffusion generative model and skeletal postures extracted from manual photography. The image template library can be composed of high-quality template images that are scored and reviewed by a professional aesthetic review team.

[0052] In one embodiment, the cloud inputs the portrait image into a multimodal recognition model, the recognition model analyzes the portrait features in the portrait image, and the recognition model outputs the portrait feature label of the portrait image.

[0053] The cloud matches the portrait feature labels output by the recognition model with the reference labels of each image template in the image template library, and maps them to the image template that best matches the portrait feature labels.

[0054] Among them, the portrait feature label is the key information extracted from the portrait picture, which is used to describe the portrait features in the portrait picture. The portrait feature label can be female, 20 years old, round face, etc.

[0055] The reference label extracts key information from each image template and is used to describe the portrait features of each image template. The reference label can be male, 30 years old, round face, etc.

[0056] The recognition model is a pre-trained deep learning model that is used to identify portrait features in portrait images and output corresponding portrait feature labels.

[0057] As described above, by inputting the portrait picture into the recognition model, obtaining the portrait feature label output by the recognition model, matching the portrait feature label with the reference label of each image template, and obtaining the image template, the recognition model can be used to accurately extract the key features in the portrait picture and match them with the reference label of the image template, thereby further ensuring a high degree of consistency between the determined image template and the portrait picture uploaded by the user.

[0058] Step 104: Fusing the features of the portrait image and the image template to generate a digital human image.

[0059] In this step, the cloud performs feature fusion of the portrait image and the image template, extracts the facial features in the portrait image, adjusts the facial features in the image template, and generates a digital human image, ensuring that the generated digital human image retains the facial features in the portrait image and conforms to the matching image template.

[0060] Among them, the digital human image is a digitized portrait image that retains the facial features in the portrait image and conforms to the matching image template.

[0061] Step 106: Use the digital human image to replace the motion capture video template corresponding to the image template to generate digital human data.

[0062] In this step, the cloud uses image synthesis technology to replace the motion capture video template corresponding to the image template with a digital human image, and combines the facial features in the digital human image with the body movements and expressions in the motion capture video to generate digital human data.

[0063] The motion capture video template is a motion capture video that is pre-shot and stored in the cloud. The ratio of the motion capture video template is consistent with that of the image template. The motion capture video template may include: video templates for specific body movements and expressions. Specific body movements may include greeting, dancing, body swaying, etc.

[0064] Digital human data is generated after portrait replacement and synthesis, and may include at least one of the following: digital human video, picture sequence. Digital human video is a video obtained by replacing facial features in motion capture video, and picture sequence is obtained by frame extraction conversion of digital human video.

[0065] In one embodiment, the cloud can use the trained action-driven big model to generate digital human data. The digital human image and the motion capture video template are input into the action-driven big model, and the action-driven big model obtains the motion capture image of each frame in the motion capture video template, extracts the facial features in the digital human image and the reference features in the motion capture image.

[0066] For each frame of motion capture image, the action-driven large model uses feature matching algorithms such as Fast Library for Approximate Nearest Neighbors (FLANN) and Random Sample Consensus (RANSAC) to match the facial features in the digital human image with the reference features of the motion capture image of the current frame, map the facial features to the matching reference features, and generate digital human data after facial feature replacement.

[0067] Among them, the motion capture image refers to a single frame image extracted from the motion capture video template, which is used to reflect the posture and position of the digital human at a specific time point. The reference feature refers to the feature extracted from the motion capture image, which is used to describe the posture and position of the digital human in the motion capture video template.

[0068] As described above, by obtaining the motion capture picture of each frame in the motion capture video template, extracting the portrait feature points in the digital human picture and the reference feature points in the motion capture picture, mapping the portrait features to the reference features, and generating digital human data, it is only necessary to replace the feature points in the motion capture video template to achieve a realistic motion synchronization effect and reduce the customization cost of the digital human.

[0069] Step 108: Send the digital human data to the vehicle computer so that the vehicle computer displays the digital human in the digital human data.

[0070] In this step, the cloud sends the digital human data to the vehicle terminal, and the vehicle terminal displays the digital human in the digital human data on the interactive page of the vehicle terminal based on the received digital human data, presenting the digital human to the user. The cloud can also send the digital human data to the user terminal, and the user terminal displays the digital human in the digital human data on the interface.

[0071] After receiving the digital human data, the vehicle computer can preview the personalized digital human to the user on the interactive page. After confirming that the user chooses to apply, the original digital human will be replaced and the personalized digital human will be displayed.

[0072] The digital human customization method of this embodiment identifies the portrait picture uploaded by the user, determines the image template that matches the portrait picture, fuses the features of the portrait picture and the image template to generate a digital human picture, uses the digital human picture to replace the portrait of the motion capture video template corresponding to the image template, generates digital human data, and sends the digital human data to the vehicle computer end so that the vehicle computer end displays the digital human in the digital human data. The unique digital human image can be customized and displayed, thereby improving the user's personalized experience of the digital human. In addition, there is no need to perform complex processes such as making 3D models and model mapping, thereby reducing the time cost and process complexity of customizing digital humans.

[0073] See also Figure 2 , Figure 2 This is a flow chart of another digital human customization method according to an exemplary embodiment of the present application. The digital human customization method can be applied to the vehicle terminal, and the digital human customization method can include the following steps:

[0074] Step 202: receiving digital human data sent by the cloud; wherein the digital human data is generated by fusing features of the portrait image uploaded by the user with the matching image template, and performing portrait replacement on the motion capture video template corresponding to the image template.

[0075] In this step, the user uploads the portrait picture through the user end, and the cloud matches the image template library based on the portrait features in the portrait picture to determine the image template that best matches the portrait picture.

[0076] The cloud merges the features of the portrait image with the image template to generate a digital human image that is both in line with the user's characteristics and has a certain style. The digital human image is used to replace the portrait of the motion capture video template corresponding to the image template, and the facial features in the digital human image are combined with the body movements and expressions in the motion capture video to generate digital human data.

[0077] The cloud sends digital human data such as digital human videos and picture sequences to the vehicle computer, and the vehicle computer receives the digital human data sent by the cloud.

[0078] Step 204: Display the digital human in the digital human data.

[0079] In this step, the vehicle computer displays the digital human in the digital human data on the interactive interface.

[0080] The digital human customization method of this embodiment receives digital human data sent from the cloud and displays the digital human in the digital human data. Since the digital human data is generated by integrating the features of the portrait image uploaded by the user with the matching image template and replacing the portrait with the motion capture video template corresponding to the image template, a unique digital human image can be customized and displayed, thereby improving the user's personalized experience of the digital human. In addition, there is no need to perform complex processes such as making 3D models and model mapping, thereby reducing the time cost and process complexity of customizing digital humans.

[0081] In the above-mentioned embodiments, it is introduced that a digital human image is generated by using a personalized portrait image and an image template, and a motion capture video template is replaced with a portrait using the digital human image to generate a customized digital human. In the following embodiments, the application scenarios of digital humans will be introduced in detail and can be applied to any of the above embodiments.

[0082] In one embodiment, the user inputs or speaks his or her questions through a touch screen, voice input, etc. The vehicle computer receives the user's question information and sends the user's question information to the cloud.

[0083] The cloud receives the question information sent by the vehicle computer. Using the large language model, the large language model deeply understands and analyzes the question information based on its huge knowledge base and training data, and generates a reply text for the question information.

[0084] The cloud uses text-to-speech technologies such as the Text-To-Speech (TTS) model to convert the reply text into reply voice. TTS technology can simulate the intonation, speed and stress of human voice, making the generated voice more realistic.

[0085] The cloud sends the generated reply voice to the vehicle computer. After receiving the reply voice, the vehicle computer drives the lip movements of the digital human in the digital human data based on the reply voice and the digital human data.

[0086] For example, the vehicle terminal can input the reply speech and the digital human data into the lip-shaped driving model, which analyzes the characteristics of the speech signal and generates a series of lip-shaped animation frames corresponding to the reply speech. The lip-shaped animation frames are used to describe the changes in the shape of the digital human's mouth from the beginning to the end of the speech. The lip-shaped animation frames are replaced with the lip shape area of ​​the digital human in the digital human data, and the lip-shaped driving model outputs the digital human data with the corresponding lip shape.

[0087] While playing the reply voice, the vehicle computer displays the digital human data with corresponding lip shapes to ensure that the lip shapes of the digital human in the digital human data completely match the voice content being played.

[0088] The question information is a question or request raised by the user to the vehicle computer, which is used to reflect the specific content that the user wants to know or execute. The reply text is the reply content in text form generated when processing the question information. The reply voice is the reply content generated based on the question information and presented in voice form.

[0089] The lip-driven model is a machine learning model that generates corresponding lip animations based on speech content. It is a terminal-side model on the vehicle side. The lip-driven model is used to analyze the characteristics of speech signals and predict and generate mouth shape changes that match the speech content.

[0090] As described above, by receiving the question information sent by the vehicle computer, using the large language model, generating the reply text of the question information, converting the reply text into reply voice, and sending the reply voice to the vehicle computer, the vehicle computer drives the digital human lip movements based on the reply voice and the digital human data, while ensuring the accuracy and fluency of the reply content, enriching the interaction form, making the interaction process more intuitive and vivid.

[0091] In the above-mentioned embodiments, the use of advanced lip-driven technology to enhance the immersion and realism of human-computer interaction is introduced, and the lip shape of the digital human is driven in real time according to the reply voice of the question information, providing users with a more natural and smooth interaction experience. In the following embodiments, the stylized rendering process of the digital human will be introduced in detail, and can be applied to any of the above embodiments.

[0092] In one embodiment, the user can select the style label of the digital human through the vehicle terminal or the user terminal by voice input, manual selection, etc. The cloud receives the style label selected by the user and determines the style guidance model corresponding to the style label according to the mapping relationship between the preset style label and the style guidance model.

[0093] The cloud uses the selected style tag and the style guidance model corresponding to the style tag as the guidance text (prompt). The digital human image and guidance text generated after feature fusion are input into the pre-trained style rendering model to obtain the digital human image that conforms to the style tag output by the style rendering model.

[0094] Among them, style labels are descriptive words that describe the visual style expected by users. Style labels are used to represent different artistic styles and emotional colors, such as: two-dimensional, cyberpunk, national trend, retro, science fiction, etc.

[0095] The guidance text is used to guide the style rendering model to generate the description text of the target style. The style guidance model is a set of predefined parameters, rule sets or machine learning models used to guide the subsequent style rendering process.

[0096] The style rendering model is an algorithm or model based on deep learning technology. It is used to accept the original digital human image and the style label specified by the user as input, and output the digital human image with the user's desired style through a series of calculations and processing processes.

[0097] As described above, by using the selected style tag and the style guidance model corresponding to the style tag as the guidance text, the digital human image and the guidance text are input into the style rendering model, so that the digital human image output by the style rendering model can closely fit the style tag selected by the user, further enhancing the user's personalized experience of the digital human.

[0098] In one embodiment, the style rendering model may include: an embedding unit, an encoding unit, and a decoding unit. Figure 3 , Figure 3 A flow chart of a stylized rendering method is shown. The stylized rendering method may include the following steps:

[0099] Step 302: The embedding unit converts the digital human image and the guidance text into first image information.

[0100] In this step, the embedding unit performs embedding encoding on the digital human image and the guidance text, and converts the digital human image and the guidance text into first image information.

[0101] Step 304: The encoding unit maps the first image information and the initialized noise information to a latent space, performs iterative noise reduction, and obtains second image information.

[0102] In this step, the encoding unit maps the first image information and the initialized noise information to a latent space, performs iterative noise reduction in the latent space, gradually reduces the noise in the image and refines the image information, and obtains the second image information.

[0103] Step 306: The decoding unit upgrades the second image information to the image space to generate a digital human image that conforms to the style label.

[0104] In this step, the decoding unit upgrades the second image information to the image space, and gradually restores the details and style features of the original digital human image through a series of inverse transformation and reconstruction operations, thereby generating a digital human image that conforms to the style label.

[0105] Among them, the embedding unit may be an Embeding layer, the encoding unit may be an Encoder, and the decoding unit may be a Decoder.

[0106] The first image information is a preliminary image representation after being processed by the embedding unit, which integrates the features of the original digital human image and the style information in the guidance text, and can provide basic data for the subsequent latent space mapping and iterative denoising process.

[0107] The second image information is an image representation obtained after iterative noise reduction and optimization in the latent space. Compared with the first image information, the second image information has purer and richer style features and is closer to the final image style to be generated. The second image information is used to generate the final digital human image that meets the user-specified style label.

[0108] As described above, the digital human image and the guidance text are converted into first image information through the embedding unit in the style rendering model, the encoding unit maps the first image information and the initialized noise information to the latent space, performs iterative denoising, and obtains second image information, and the decoding unit upgrades the second image information to the image space to generate a digital human image that conforms to the style label. The iterative denoising process in the latent space helps to generate a digital human image that is purer and conforms to the target style, which can not only retain the details and clarity of the original digital human image, but also achieve accurate rendering of the style.

[0109] In one embodiment, the style guidance model may include: a learnable re-weighting (Lora) model. The style rendering model may include: a basic model.

[0110] When generating digital human images that match style labels, the cloud can use a reweighted model to treat each layer in the neural network as a weighted feature extractor, and the weight of each layer determines the impact on the output of the style rendering model.

[0111] By using the reweighted model in the guidance text, some weights of the basic model in the style rendering model are fine-tuned to improve the style effect of the digital human image.

[0112] Among them, the reweighted model is used to fine-tune the basic model in the style rendering model. The basic model is a model obtained by training the improved latent diffusion (Scaled Diffusion Cross-Level, SDXL) model using the image-text pair dataset, which is used to enhance the effect of generating character images. The image-text pair dataset is generated by manually annotating a large number of character image pictures.

[0113] As mentioned above, by utilizing the reweighted model, the base model in the style rendering model is fine-tuned. Without changing the overall structure of the base model, the weight parameters are adjusted to adapt to different style requirements, and a variety of style transfer tasks can be flexibly responded to without retraining the entire style rendering model for each style label.

[0114] To further introduce the interactive process of digital human generation and display, Figure 4An interactive diagram of a digital human customization method is shown. Figure 4 As shown, the digital human customization method may include the following steps:

[0115] Step 402: The user terminal sends the portrait image to the cloud.

[0116] In this step, the user uploads the portrait image through the user terminal, and the user terminal sends the portrait image to the cloud.

[0117] Step 404: The cloud inputs the portrait image into the recognition model to obtain the portrait feature label.

[0118] In this step, the cloud inputs the portrait image into the recognition model, and the recognition model analyzes the portrait features in the portrait image to obtain the portrait feature label output by the recognition model.

[0119] Step 406: The cloud matches the portrait feature label with the reference label to obtain an image template.

[0120] In this step, the cloud matches the portrait feature labels with the reference labels and determines the best matching image template in the image template library.

[0121] Step 408: The cloud performs feature fusion of the portrait image and the image template to generate a digital human image.

[0122] In this step, the cloud merges the facial features of the portrait image and the image template to generate a digital human image that retains the facial features in the portrait image and conforms to the matching image template.

[0123] Step 410: The cloud uses the style label and the style guidance model as guidance text.

[0124] In this step, the cloud uses the selected style tag and the style guidance model corresponding to the style tag as guidance text.

[0125] Step 412: The cloud obtains a digital human image that matches the style tag.

[0126] In this step, the cloud inputs the digital human image and the guidance text into the style rendering model, and the embedding unit in the style rendering model converts the digital human image and the guidance text into the first image information. The encoding unit maps the first image information and the initialized noise information to the latent space, performs iterative noise reduction, gradually reduces the noise in the image and refines the image information, and obtains the second image information.

[0127] The decoding unit upgrades the second image information to the image space, restores the details and style features of the original digital human image, and generates a digital human image that conforms to the style label.

[0128] Step 414: Send the digital human image to the user terminal.

[0129] In this step, the user terminal previews the received digital human picture in the interface, displays the static digital human image in the digital human picture to the user, and receives the user's feedback information on the digital human picture.

[0130] Step 416: The cloud determines whether the user is satisfied with the digital human image.

[0131] In this step, the user terminal sends the user's feedback information to the cloud, and the cloud determines whether the user is satisfied with the digital human image based on the positive or negative feedback information.

[0132] If it is determined that the user is satisfied with the digital human image, step 418 is executed;

[0133] If it is determined that the user is not satisfied with the digital human image, step 406 is executed to remove the selected image template and redetermine the best matching image template.

[0134] Step 418: The cloud replaces the human figure in the motion capture video template to generate a picture sequence.

[0135] In this step, the cloud obtains the motion capture image of each frame in the motion capture video template corresponding to the image template, extracts the facial features in the digital human image and the reference features in the motion capture image, maps the facial features to the reference features, generates a digital human video, extracts frames from the digital human video, and obtains a sequence of images containing the digital human.

[0136] The cloud can scale, expand, or crop the image sequence to obtain a standardized image sequence.

[0137] Step 420: The cloud sends the image sequence to the vehicle computer.

[0138] Step 422: The vehicle terminal displays the digital human in the picture sequence.

[0139] Step 424: The vehicle computer sends a question message to the cloud.

[0140] In this step, the user inputs question information to the vehicle computer by voice, and the vehicle computer sends the question information to the cloud.

[0141] Step 426: The cloud generates a reply voice corresponding to the question information based on the question information.

[0142] In this step, the cloud inputs the question information into the large language model, obtains the reply text generated by the large language model, and the cloud converts the reply text into reply speech.

[0143] Step 428: The cloud sends a reply voice message to the vehicle computer.

[0144] Step 430: The vehicle terminal drives the digital human by lip-syncing based on the reply voice and the digital human data.

[0145] In this step, the vehicle computer uses the lip-shaped driving model on the terminal side to generate a sequence of pictures with corresponding lip shapes based on the reply voice and digital human data, and displays the sequence of pictures with corresponding lip shapes while playing the reply voice.

[0146] Figure 5 1 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. The electronic device may be, for example, a mobile phone, a computer, a digital broadcast terminal, a message transceiver, a game console, a tablet device, a personal digital assistant, a server, a smart home appliance, a car computer, etc. Figure 5 At the hardware level, the electronic device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other services. The processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it, forming a cloud or vehicle terminal at the logical level. Of course, in addition to software implementations, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0147] Figure 6 This is a block diagram of a cloud according to an exemplary embodiment of the present application. Figure 6 , the cloud may include: an identification module 602, a fusion module 604, a generation module 606 and a processing module 608, wherein:

[0148] The recognition module 602 is used to recognize the portrait picture uploaded by the user and determine the image template matching the portrait picture;

[0149] The fusion module 604 is used to fuse the features of the portrait image with the image template to generate a digital human image;

[0150] The generating module 606 is used to use the digital human image to replace the motion capture video template corresponding to the image template with the human image to generate digital human data;

[0151] The processing module 608 is used to send the digital human data to the vehicle terminal, so that the vehicle terminal displays the digital human in the digital human data.

[0152] In one example, the recognition module 602, when used to recognize a portrait picture uploaded by a user and determine an image template that matches the portrait picture, includes: inputting the portrait picture into a recognition model to obtain a portrait feature label output by the recognition model; matching the portrait feature label with a reference label of each image template to obtain the image template.

[0153] In one example, the generation module 606, before using the digital human image to replace the motion capture video template corresponding to the image template to generate digital human data, also includes: using the selected style tag and the style guidance model corresponding to the style tag as guidance text; inputting the digital human image and the guidance text into the style rendering model to obtain the digital human image output by the style rendering model that conforms to the style tag.

[0154] In one example, the style rendering model includes: an embedding unit, an encoding unit and a decoding unit; the generation module 606, when used to input the digital human image and the guidance text into the style rendering model to obtain the digital human image that conforms to the style label output by the style rendering model, includes: the embedding unit converts the digital human image and the guidance text into first image information; the encoding unit maps the first image information and the initialized noise information to a latent space, performs iterative denoising, and obtains second image information; the decoding unit upgrades the second image information to an image space to generate a digital human image that conforms to the style label.

[0155] In one example, the style guidance model includes: a reweighted model; the reweighted model is used to fine-tune the base model in the style rendering model.

[0156] In one example, the generation module 606, when used to use the digital human image to replace the motion capture video template corresponding to the image template to generate digital human data, includes: obtaining a motion capture image of each frame in the motion capture video template; extracting the portrait features in the digital human image and the reference features in the motion capture image; mapping the portrait features to the reference features to generate the digital human data.

[0157] In one example, the processing module 608 is also used to receive question information sent by the vehicle terminal; generate a reply text for the question information using a large language model; convert the reply text into a reply voice; and send the reply voice to the vehicle terminal so that the vehicle terminal drives the digital human lip movements based on the reply voice and the digital human data.

[0158] Figure 7is a block diagram of a vehicle terminal according to an exemplary embodiment of the present application. Figure 7 The vehicle terminal may include: a receiving module 702 and a display module 704, wherein:

[0159] The receiving module 702 is used to receive digital human data sent by the cloud; wherein the digital human data is generated by fusing the features of the portrait picture uploaded by the user with the matching image template, and replacing the portrait with the motion capture video template corresponding to the image template;

[0160] The display module 704 is used to display the digital human in the digital human data.

[0161] In one example, the display module 704 is also used to send the user's question information to the cloud, so that the cloud uses a large language model to generate a reply text for the question information and convert the reply text into a reply voice; receive the reply voice sent by the cloud; and lip-drive the digital human based on the reply voice and the digital human data.

[0162] The implementation process of the functions and effects of each unit in the above-mentioned cloud and vehicle-mounted terminals is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0163] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0164] In an exemplary embodiment, a non-temporary computer-readable storage medium including instructions is also provided, such as a memory including instructions. The above instructions can be executed by a processor to implement a method executed by the cloud or by the vehicle end as in any of the above embodiments.

[0165] The non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and the present application does not limit this.

[0166] In an exemplary embodiment, a computer program product including a computer program / instructions is also provided. The computer program / instructions can be executed by a processor to implement a method executed by the cloud or by the vehicle terminal as in any of the above embodiments.

[0167] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0169] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A digital human customization method, characterized in that: The method is applied to the cloud, and the method includes: Identify the portrait picture uploaded by the user and determine the image template that matches the portrait picture; Fusing the portrait image with the image template to generate a digital human image; Using the digital human image, performing a human portrait replacement on the motion capture video template corresponding to the image template to generate digital human data; The digital human data is sent to the vehicle terminal, so that the vehicle terminal displays the digital human in the digital human data.

2. The method according to claim 1, characterized in that: The step of identifying the portrait picture uploaded by the user and determining the image template matching the portrait picture includes: Inputting the portrait image into a recognition model to obtain a portrait feature label output by the recognition model; The portrait feature label is matched with the reference label of each image template to obtain the image template.

3. The method according to claim 1, characterized in that Before using the digital human image to replace the motion capture video template corresponding to the image template with a human portrait to generate digital human data, the method further includes: Using the selected style tag and the style guidance model corresponding to the style tag as guidance text; The digital human image and the guidance text are input into a style rendering model to obtain a digital human image output by the style rendering model that conforms to the style label.

4. The method according to claim 3, characterized in that The style rendering model includes: an embedding unit, an encoding unit and a decoding unit; The step of inputting the digital human image and the guidance text into a style rendering model to obtain a digital human image output by the style rendering model that conforms to the style label includes: The embedding unit converts the digital human image and the guidance text into first image information; The encoding unit maps the first image information and the initialized noise information to a latent space, performs iterative noise reduction, and obtains second image information; The decoding unit upgrades the second image information to an image space to generate a digital human image that conforms to the style label.

5. The method according to claim 3, characterized in that: The style guidance model includes: a reweighted model; the reweighted model is used to fine-tune the basic model in the style rendering model.

6. The method according to claim 1, characterized in that The step of using the digital human image to replace the motion capture video template corresponding to the image template with a human portrait to generate digital human data includes: Obtaining a motion capture image of each frame in the motion capture video template; Extracting portrait features from the digital human image and reference features from the motion capture image; The portrait feature is mapped to the reference feature to generate the digital human data.

7. The method according to claim 1, characterized in that The method further comprises: Receiving question information sent by the vehicle terminal; Generate a reply text for the question information using a large language model; Converting the reply text into reply voice; The reply voice is sent to the vehicle terminal, so that the vehicle terminal drives the digital human lip movements based on the reply voice and the digital human data.

8. A digital human customization method, characterized in that: The method is applied to a vehicle terminal, and the method includes: Receiving digital human data sent by the cloud; wherein the digital human data is generated by fusing features of a portrait picture uploaded by a user with a matching image template, and performing portrait replacement on a motion capture video template corresponding to the image template; The digital human in the digital human data is displayed.

9. The method according to claim 8, characterized in that The method further comprises: Sending the user's question information to the cloud, so that the cloud generates a reply text of the question information by using a large language model, and converts the reply text into a reply voice; Receiving the reply voice sent by the cloud; Based on the reply voice and the digital human data, the digital human is lip-driven.

10. A cloud, characterized in that: The cloud comprises: A recognition module, used to recognize the portrait picture uploaded by the user and determine the image template that matches the portrait picture; A fusion module, used to fuse the features of the portrait image with the image template to generate a digital human image; A generating module, configured to use the digital human image to replace the motion capture video template corresponding to the image template with the human image, thereby generating digital human data; The processing module is used to send the digital human data to the vehicle terminal so that the vehicle terminal displays the digital human in the digital human data.

11. A vehicle terminal, characterized in that: The vehicle terminal includes: A receiving module, used to receive digital human data sent from the cloud; wherein the digital human data is generated by fusing the features of the portrait image uploaded by the user with the matching image template, and replacing the portrait with the motion capture video template corresponding to the image template; The display module is used to display the digital human in the digital human data.

12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 9 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the method described in any one of claims 1 to 7 or the method described in any one of claims 8 to 9 is implemented.

14. A computer program product having a computer program / instructions stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 7 or the method according to any one of claims 8 to 9 is implemented.

Images