A virtual image generation method and device, a terminal device, and a storage medium

Abstract

The application relates to the technical field of image processing, and discloses a virtual image generation method and device, a terminal device and a storage medium. The method uses image features of original images corresponding to virtual images, generates corresponding videos through an image diffusion model, and then inputs the videos into a coding and decoding network included in the image diffusion model and a ControlNet network used for controlling the image diffusion model for processing. After being processed by the coding layer, the intermediate layer and the decoding layer of the coding and decoding network and the coding layer, the intermediate layer and the zero convolution layer of the ControlNet network, a processed video file is finally output by the decoding layer, and the video file can be regarded as a virtual image with original image features. Through the arrangement, a user can select original images corresponding to virtual images to be generated according to preferences, and the obtained virtual images are no longer fixed images, and can meet personalized demands of the user.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, terminal device, and storage medium for generating virtual images. Background Technology

[0002] With the continuous development of artificial intelligence technology, virtual avatars such as intelligent voice assistants have been widely used in various fields. For example, in the automotive field, in-vehicle voice assistants can provide drivers with services such as voice navigation, vehicle control, and music playback, thereby effectively improving the user's driving experience. However, current in-vehicle voice assistants and other virtual avatars are generally pre-set fixed images, which cannot meet the personalized needs of users. Summary of the Invention

[0003] In view of this, embodiments of this application provide a method, apparatus, terminal device, and storage medium for generating virtual avatars, which can meet users' personalized needs for virtual avatars.

[0004] The first aspect of this application provides a method for generating a virtual avatar, comprising:

[0005] Obtain the original image corresponding to the virtual avatar;

[0006] A first video corresponding to the original image is generated using a trained image diffusion model.

[0007] The first video is input to the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing. The encoding / decoding network contains an encoding layer, an intermediate layer and a decoding layer in sequence. The ControlNet network contains the same encoding layer, the same intermediate layer and a zero convolutional layer as the encoding / decoding network in sequence. The output of the zero convolutional layer is connected to the input of the decoding layer.

[0008] The generated virtual image is output through the decoding layer.

[0009] This embodiment utilizes the image features of the original image corresponding to the virtual avatar to generate a corresponding video through an image diffusion model. The video is then input into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing. After passing through the encoding, intermediate, and decoding layers of the encoding / decoding network, and the encoding, intermediate, and zero-convolutional layers of the ControlNet network, the final video file is output by the decoding layer of the encoding / decoding network. This video file can be considered as a virtual avatar with the features of the original image. With this setup, users can select the original image corresponding to the desired virtual avatar according to their personal preferences, such as an image of a male character or an animated character. After inputting the original image, the corresponding virtual avatar is automatically output. The resulting virtual avatar is no longer a fixed image and can meet the user's personalized needs.

[0010] In one implementation of this application, generating a first video corresponding to the original image using a trained image diffusion model may include:

[0011] The original image is semantically segmented using the SegNet network included in the image diffusion model to obtain a feature image;

[0012] The image diffusion model uses a semantic learning network to convert feature images into target text.

[0013] The target image is obtained by performing image diffusion processing on the target text and feature image through the image diffusion network contained in the image diffusion model.

[0014] The text-to-speech network included in the image diffusion model converts the target text into the target audio.

[0015] The target image and target audio are fused to obtain the first video.

[0016] Furthermore, the SegNet network structure can contain multiple network layer groups, each containing at least one convolutional layer and at least one pooling layer, with a skip connection added between the first convolutional layer and the last pooling layer in each network layer group.

[0017] In one implementation of this application, the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network for controlling the image diffusion model for processing may include:

[0018] Obtain the preset image template;

[0019] The first video and the image template are used as input data and are respectively fed into the encoding / decoding network and the ControlNet network for processing.

[0020] In another implementation of this application, the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network for controlling the image diffusion model for processing may include:

[0021] Retrieve the prompt text corresponding to the virtual avatar;

[0022] The first video and the prompt text are used as input data and are fed into the encoding / decoding network and ControlNet network for processing, respectively.

[0023] In another implementation of this application, the virtual avatar is the vehicle's in-vehicle voice assistant avatar; the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing may include:

[0024] Obtain vehicle parameters during the vehicle's operation;

[0025] The first video and vehicle parameters are used as input data and are respectively fed into the encoding / decoding network and the ControlNet network for processing.

[0026] In one implementation of this application, the virtual image generated by outputting through the decoding layer may include:

[0027] The second video is output through the decoding layer;

[0028] For each frame of the second video, the range of the image containing the virtual image's edge is obtained; each pixel within the range is used as input to a distance measurement algorithm to increase the distance between pixels within the range that belong to the virtual image's edge and pixels that do not belong to the virtual image's edge, thus obtaining the processed second video.

[0029] The processed second video is then identified as the generated virtual avatar.

[0030] A second aspect of this application provides a virtual avatar generation apparatus, comprising:

[0031] The original image acquisition module is used to acquire the original image corresponding to the virtual image;

[0032] The video generation module is used to generate a first video corresponding to the original image using a trained image diffusion model;

[0033] The video input module is used to input the first video to the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing; wherein, the encoding / decoding network includes an encoding layer, an intermediate layer and a decoding layer in sequence, and the ControlNet network includes the same encoding layer, the same intermediate layer and a zero convolutional layer as the encoding / decoding network in sequence, and the output of the zero convolutional layer is connected to the input of the decoding layer.

[0034] The virtual avatar output module is used to output the generated virtual avatar through the decoding layer.

[0035] A third aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the virtual avatar generation method provided in the first aspect of this application.

[0036] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating a virtual image as provided in the first aspect of this application.

[0037] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the virtual image generation method provided in the first aspect of this application.

[0038] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a flowchart of a method for generating a virtual image provided in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram of the structure of an encoding / decoding network and a ControlNet network provided in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of an overall process for a virtual image generation method provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the data processing flow for generating the first video;

[0043] Figure 5 This is a structural framework diagram of a virtual image generation device provided in an embodiment of this application;

[0044] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0045] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0046] Currently, when people drive, they can utilize in-vehicle voice assistants to provide services such as voice navigation, vehicle control, and music playback, effectively improving the user's driving experience. However, virtual avatars such as in-vehicle voice assistants are generally pre-set fixed images, failing to meet users' personalized needs. To address this issue, this application provides a method, apparatus, terminal device, and storage medium for generating virtual avatars. Personalized virtual avatars can be generated through image processing algorithms, thereby satisfying users' personalized needs for virtual avatars. For more specific technical implementation details of this application's embodiments, please refer to the various embodiments described below.

[0047] It should be understood that the execution subject of the various method embodiments of this application can be various types of terminal devices or servers, such as mobile phones, tablets, wearable devices, vehicle controllers, in-vehicle terminals, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), personal digital assistants (PDAs), etc. The embodiments of this application do not impose any restrictions on the specific type of terminal device and server.

[0048] Please see Figure 1 This application illustrates a method for generating a virtual avatar, comprising:

[0049] 101. Obtain the original image corresponding to the virtual avatar;

[0050] The execution entity in this application embodiment can generally be a vehicle-mounted terminal or a user-used mobile phone / tablet or other terminal device. First, the original image corresponding to the virtual avatar is acquired. Users can select the desired virtual avatar according to their personalized needs, such as a mature male avatar, a young female avatar, or a cartoon animal avatar, etc. In practice, the terminal device can pre-collect image and voice data corresponding to each different virtual avatar (e.g., different genders, ages, skin colors, and hairstyles), perform certain preprocessing on the image and voice data, such as data cleaning, data standardization, and handling missing values, and then store the preprocessed image and voice data. When the user selects the desired virtual avatar, the terminal device can search for the image data of that virtual avatar and obtain multiple frames as the original image.

[0051] 102. Using the trained image diffusion model, generate the first video corresponding to the original image;

[0052] After obtaining the original image, it can be input into a trained image diffusion model for processing. After feature extraction and image diffusion by the image diffusion model, a video with the features of the original image can be output, referred to here as the first video. The image diffusion model can employ various existing image diffusion processing algorithms, such as the Diffusion algorithm, Stable Diffusion algorithm, and Guided Diffusion algorithm, etc.

[0053] In one implementation of this application, generating a first video corresponding to the original image using a trained image diffusion model may include:

[0054] (1) The original image is processed by semantic segmentation through the SegNet network contained in the image diffusion model to obtain the feature image;

[0055] (2) The feature image is converted into target text through the semantic learning network contained in the image diffusion model;

[0056] (3) The target image is obtained by performing image diffusion processing on the target text and feature image through the image diffusion network contained in the image diffusion model;

[0057] (4) Convert the target text into target audio using the text-to-speech network included in the image diffusion model;

[0058] (5) The target image and target audio are fused to obtain the first video.

[0059] The image diffusion model designed in this application embodiment can include multiple sub-networks such as a SegNet network, a semantic learning network, an image diffusion network, a text-to-speech network, and an encoding / decoding network described later. After the original image is input into the image diffusion model, the SegNet network first performs image semantic segmentation processing on the original image to extract the corresponding feature images. It is important to note that, unlike the conventional Stable Difussion model which uses a U-Net network for feature extraction, this application embodiment extends the network architecture of the Stable Difussion model by uniquely introducing the SegNet network into the backbone structure of the Stable Difussion model, that is, replacing the U-Net network with the SegNet network for feature extraction. Considering that the Stable Difussion model requires the network's input and output dimensions to be the same, and the SegNet network, while meeting this requirement, also possesses excellent image semantic segmentation capabilities, this application embodiment utilizes the SegNet network for image semantic segmentation processing of the original image, which can not only successfully extract the corresponding feature images but also significantly improve the image semantic segmentation effect.

[0060] Furthermore, the SegNet network structure can contain multiple network layer groups, each containing at least one convolutional layer and at least one pooling layer, with a skip connection added between the first convolutional layer and the last pooling layer in each network layer group.

[0061] Generally, to obtain a better-performing neural network model, the number of training iterations needs to be set to a relatively large value. However, this can lead to common problems such as vanishing and exploding gradients. To address this issue, this application improves the network structure of the SegNet network. Specifically, the convolutional and pooling layers of the SegNet network are grouped into multiple network layer groups. Each network layer group contains at least one convolutional layer and at least one pooling layer. Furthermore, a skip connection is added between the first convolutional layer and the last pooling layer in each network layer group. By adding skip connections, similar to structures like residual networks, the vanishing and exploding gradient problems can be avoided as the model undergoes deep iterative training, while simultaneously allowing the model to achieve faster convergence and better performance.

[0062] After obtaining the feature images, a semantic learning network is used to convert them into natural language text, denoted as the target text. Here, a natural language processing model based on recurrent neural networks can be used as the semantic learning network to convert the extracted feature image information into natural language text, such as textual information about gender, age, skin color, hairstyle, actions, expressions, and emotions. The specific principles of using a natural language processing model to convert feature images into natural language text can be found in existing technologies and will not be elaborated upon here.

[0063] After obtaining the target text, an image diffusion network is used to perform image diffusion processing on the target text and feature images to obtain the target image. For example, an image diffusion network based on the Stable Difussion algorithm can be trained. Both the target text and feature images are input into this image diffusion network for image diffusion processing. Referring to the principle of generating images using text and feature images in the Stable Difussion algorithm, the output target image can be obtained. This target image contains both the image features of the original image and the text features of the target text.

[0064] On the other hand, the target text can be converted into the target audio through a text-to-speech network. Here, technologies such as text-to-speech (TTS) can be used to convert the target text into the corresponding audio and represent it with the target audio.

[0065] Finally, the target image and target audio can be fused to obtain the first video. It can be understood that the original image can contain multiple frames of visual images, and each frame can obtain its corresponding target image and target audio. By fusing the target images and target audio of all the visual images and then stitching them together, a video file with audio can be obtained, which is the first video.

[0066] In one implementation of this application, the method may further include:

[0067] Obtain the first audio file corresponding to the virtual avatar;

[0068] The process of fusing the target image and target audio to obtain the first video may include:

[0069] The target image, target audio, and first audio are fused to obtain the first video.

[0070] As mentioned earlier, each virtual avatar can pre-store corresponding image and voice data, with the voice data containing the audio for each avatar. When a user selects the desired virtual avatar, the corresponding audio is retrieved, denoted as the "first audio." For example, if the user wants a mature man, the first audio is the voice of a mature man; if the user wants a young woman, the first audio is the voice of a young woman, and so on. During video synthesis, the first audio and the target audio are first superimposed to obtain the superimposed audio. Then, the superimposed audio is fused with the target image to finally synthesize the first video. Through this setup, the final generated virtual avatar will have a personalized voice matching the avatar, thereby further enhancing the user's viewing experience.

[0071] In one implementation of this application, after converting the feature image into target text, the method may further include:

[0072] The target text is subjected to sentiment analysis to obtain sentiment tags.

[0073] The process of fusing the target image and target audio to obtain the first video may include:

[0074] (1) Obtain the second audio corresponding to the emotion tag;

[0075] (2) The target image, target audio and second audio are fused to obtain the first video.

[0076] To further enhance the intelligence and personalization of virtual avatars by combining their voices with emotions, text sentiment analysis can be performed on the target text to obtain sentiment tags, such as happy, joyful, excited, sad, and disappointed. This can be achieved using a neural network with a memory-based attention mechanism to analyze the target text and obtain the corresponding sentiment tags. The specific principle of using an attention mechanism neural network to analyze text sentiment and obtain corresponding sentiment tags can be found in existing technologies and will not be elaborated here. The terminal device can pre-store the audio corresponding to each different sentiment tag; for example, the happy tag corresponds to an audio with a cheerful tone, the sad tag corresponds to an audio with a crying tone, and so on. After performing text sentiment analysis on the target text to obtain the sentiment tag, the audio corresponding to that sentiment tag can be found and represented by a second audio. Then, the target image, the target audio, and the second audio are fused to obtain the first video. Specifically, the second audio and the target audio can be superimposed first to obtain the superimposed audio, and then the superimposed audio and the target image are fused together to finally synthesize the first video. With this setting, the voice of the final virtual character will be associated with its emotions. For example, if the emotion tag obtained through text sentiment analysis is "happy", then the voice of the final virtual character will be cheerful.

[0077] 103. The first video is input into the encoding / decoding network contained in the image diffusion model and the ControlNet network used to control the image diffusion model for processing;

[0078] After obtaining the initial video, optimization can be performed to generate a realistic and personalized virtual avatar. Specifically, the initial video can be input into both the encoding / decoding network within the image diffusion model and the ControlNet network used to control the image diffusion model. ControlNet, an extension of Stable Diffusion developed by Stanford University researchers, enables creators to easily control objects in AI images and videos. It controls image generation based on various conditions such as edge detection, sketching, or human pose, and can be summarized as a simple, stable diffusion fine-tuning method.

[0079] The codec network can sequentially include an encoding layer, an intermediate layer, and a decoding layer. The ControlNet network can sequentially include the same encoding layer, intermediate layer, and zero-convolutional layer as the codec network, with the output of the zero-convolutional layer connected to the input of the decoding layer. For example... Figure 2 The diagram shown is a structural schematic of an encoding / decoding network and a ControlNet network provided in an embodiment of this application. Figure 2In the middle, on the left is the encoding and decoding network of the image diffusion model, which consists of multiple encoding layers, an intermediate layer, and multiple decoding layers corresponding to the encoding layers. Figure 2 To the right is the ControlNet network. To maintain the stability of the image diffusion model's output, only minor adjustments are made to the output. Here, the ControlNet network includes the same encoding layer and intermediate layers as the encoding / decoding network. The ControlNet network lacks a decoding layer; instead, it uses multiple zero-convolutional layers. These zero-convolutional layers are 1x1 convolutions with zero initial weights and biases. Using zero-convolutional layers maintains the consistency between the image diffusion model and the training data. For specific data flow details, please refer to... Figure 2 As indicated by the arrows, the first video stream is input to both the codec network and the ControlNet network. One stream of data is processed sequentially through the encoding layer, intermediate layer, and decoding layer of the codec network. The other stream of data is processed sequentially through the encoding layer, intermediate layer, and zero-convolutional layer of the ControlNet network. The data processed by each zero-convolutional layer is then input to the corresponding decoding layer of the codec network. Finally, the last decoding layer of the codec network outputs the final data. This final data is essentially also a video file, which can be viewed as a virtual image with the features of the original image.

[0080] In one implementation of this application, the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network for controlling the image diffusion model for processing may include:

[0081] (1) Obtain the preset image template;

[0082] (2) The first video and the image template image are used as input data and are respectively input to the encoding / decoding network and the ControlNet network for processing.

[0083] When generating a virtual avatar based on the first video, a preset avatar template image can be obtained. Then, the avatar template image and the first video are fused using an encoding / decoding network and a ControlNet network. This involves using ControlNet technology to analyze and process the image data, ultimately creating a realistic and personalized virtual avatar. Specifically, users can have personalized requirements for their virtual avatars, such as gender, appearance, age, style, actions, expressions, clothing, hairstyle, and accessories. For each personalized requirement, a corresponding avatar template image can be constructed and stored separately, such as avatar template images for different genders and appearances, different ages and styles, different clothing and accessories, and different actions and expressions. Additionally, users can customize the avatar template image; for example, they can upload their own or someone else's photo as the avatar template image. The generated virtual avatar will then include the user's or someone else's physical characteristics. After obtaining the avatar template image, both the avatar template image and the first video can be used as input data, processed by the encoding / decoding network and ControlNet network respectively, ultimately generating the corresponding virtual avatar, which will carry the personalized features of the avatar template image.

[0084] In another implementation of this application, the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network for controlling the image diffusion model for processing may include:

[0085] (1) Obtain the prompt text corresponding to the virtual avatar;

[0086] (2) The first video and the prompt text are used as input data and are respectively input to the encoding / decoding network and the ControlNet network for processing.

[0087] In another implementation, users can pre-set the corresponding prompt text for each virtual avatar. For example, if the desired virtual avatar is a car voice assistant, commonly used prompts for car voice assistants can be obtained as the prompt text. When generating a virtual avatar using the first video, both the first video and the prompt text can be used as input data, respectively fed into the encoding / decoding network of the image diffusion model and the ControlNet network for processing. This setup ensures that the output virtual avatar also incorporates the characteristics of the prompt text, allowing users to more intuitively see the corresponding prompts.

[0088] In another implementation of this application, the virtual avatar is the vehicle's in-vehicle voice assistant avatar; the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing may include:

[0089] (1) Obtain vehicle parameters during the driving process;

[0090] (2) The first video and vehicle parameters are used as input data and are respectively input to the encoding / decoding network and ControlNet network for processing.

[0091] If a user wants to generate a virtual avatar that resembles a vehicle's in-vehicle voice assistant, vehicle parameters collected during driving can be obtained and incorporated into the generated avatar. Specifically, the in-vehicle terminal can acquire various vehicle parameters during operation, such as location, speed, acceleration, and model. When generating the virtual avatar using the first video, both the first video and the vehicle parameters can be used as input data, processed separately by the image diffusion model's encoding / decoding network and the ControlNet network. This setup allows the output virtual avatar to include vehicle parameter information such as speed. For example, this vehicle parameter information can be displayed in the vicinity of the final generated in-vehicle voice assistant avatar, allowing users to simultaneously view the current vehicle parameters while viewing the avatar, thus enhancing the driving experience.

[0092] 104. The virtual image generated by the output of the decoding layer of the encoding / decoding network.

[0093] After the first video is input into the encoding / decoding network of the image diffusion model and the ControlNet network used to control the image diffusion model, the processed video file can be output through the decoding layer of the encoding / decoding network. This video file can be regarded as a generated virtual avatar. The generated virtual avatar can be displayed to users or application systems for subsequent analysis and processing. For example, the generated virtual avatar can be displayed on the screen of an in-vehicle terminal or the screen of a user's mobile phone. Users can evaluate and provide feedback on the generated virtual avatar, including aspects such as the avatar's realism, its individuality, and user opinions.

[0094] In one implementation of this application, the virtual image generated by outputting through the decoding layer may include:

[0095] (1) Output the second video through the decoding layer;

[0096] (2) For each frame of the second video, obtain the range of the image containing the virtual image's edge; use each pixel in the range as the input of the distance measurement algorithm to perform distance measurement processing, so as to increase the distance between the pixels in the range that belong to the virtual image's edge and the pixels that do not belong to the virtual image's edge, thereby obtaining the processed second video.

[0097] (3) The processed second video is identified as the generated virtual image.

[0098] Assuming the video file output by the decoding layer is called the second video, it can be directly used as the output of the generated virtual avatar. Alternatively, if the user is more concerned with the details of the edges of the generated virtual avatar, distance metric learning can be introduced to process the second video. This improves the accuracy of pixel calculations for the virtual avatar's edges, resulting in a more realistic and natural overall presentation. Specifically, for each frame of the second video, the region containing the virtual avatar's edges can be identified. Each pixel within this region is then used as input to a distance metric algorithm to increase the distance between pixels belonging to the virtual avatar's edges and those not within the edges, thus obtaining the processed second video. In other words, pixels belonging to the virtual avatar's edges are grouped together, while pixels not belonging to the edges are pushed further apart. This allows for the learning of more realistic and accurate edge information, resulting in a clearer and more vivid virtual avatar. The second video processed by the distance metric algorithm can then be used as the final virtual avatar output.

[0099] This embodiment utilizes the image features of the original image corresponding to the virtual avatar to generate a corresponding video through an image diffusion model. The video is then input into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing. After passing through the encoding, intermediate, and decoding layers of the encoding / decoding network, and the encoding, intermediate, and zero-convolutional layers of the ControlNet network, the final video file is output by the decoding layer of the encoding / decoding network. This video file can be considered as a virtual avatar with the features of the original image. With this setup, users can select the original image corresponding to the desired virtual avatar according to their personal preferences, such as an image of a male character or an animated character. After inputting the original image, the corresponding virtual avatar is automatically output. The resulting virtual avatar is no longer a fixed image and can meet the user's personalized needs.

[0100] like Figure 3 The diagram shown is a schematic representation of an overall process for a virtual avatar generation method provided in an embodiment of this application. Figure 3 The process begins with data acquisition, which involves collecting image and audio data corresponding to different virtual avatars. Next, preprocessing operations are performed on the image and audio data, such as data cleaning, standardization, and handling of missing values. Then, the image data is input into a semantic feature extraction module for processing, yielding feature images and converted target text. Following this, the data is fused using the Stable Difussion algorithm to generate the first video. Next, the first video and the avatar template image are fused using an encoding / decoding network and a ControlNet network to generate the final virtual avatar. Finally, the generated virtual avatar can be optimized and displayed.

[0101] like Figure 4 The diagram shown illustrates the data processing flow for generating the first video. Figure 4 The input consists of multiple frames of original images corresponding to the virtual avatar. A feature extraction network, such as a convolutional neural network or SegNet, extracts feature images from these original images. Next, the feature images are input to a semantic learning module, which consists of an image decoder and a text decoder. These decoders convert the feature images into corresponding text. The converted text, along with the feature images, is then input into a diffusion network for image diffusion processing to generate the target image. Furthermore, the converted text can be converted into corresponding target audio using a text-to-speech module. Finally, the target image and target audio are combined to form the first output video.

[0102] In summary, the embodiments of this application can utilize Stable Diffusion combined with ControlNet to achieve personalized virtual avatar generation, which has the following advantages: (1) More realistic model effect: The virtual avatar generation system based on Stable Diffusion and ControlNet uses deep neural networks, which can better capture the complexity and diversity of avatar generation; (2) Higher degree of personalization: The virtual avatar generation system based on Stable Diffusion and ControlNet can generate personalized virtual avatars according to user preferences and needs, for example, it can generate personalized avatars based on photos uploaded by users or avatar templates selected; (3) Faster speed: The virtual avatar generation system based on Stable Diffusion and ControlNet uses deep neural networks, which can accelerate the virtual avatar generation process through parallel computing; (4) Stronger system scalability: The virtual avatar generation system based on Stable Diffusion and ControlNet can generate new samples through pre-trained models and transfer learning, thereby improving the scalability of the system.

[0103] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0104] The above mainly describes a method for generating virtual images. The following will describe a device for generating virtual images.

[0105] Please see Figure 5 One embodiment of a virtual avatar generation device in this application includes:

[0106] The original image acquisition module 501 is used to acquire the original image corresponding to the virtual image;

[0107] The video generation module 502 is used to generate a first video corresponding to the original image using a trained image diffusion model;

[0108] The video input module 503 is used to input the first video to the encoding and decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing; wherein, the encoding and decoding network includes an encoding layer, an intermediate layer and a decoding layer in sequence, and the ControlNet network includes the same encoding layer, the same intermediate layer and a zero convolutional layer as the encoding and decoding network in sequence, and the output of the zero convolutional layer is connected to the input of the decoding layer.

[0109] The virtual avatar output module 504 is used to output the generated virtual avatar through the decoding layer.

[0110] In one implementation of this application, the video generation module may include:

[0111] The feature extraction unit is used to perform image semantic segmentation processing on the original image through the SegNet network included in the image diffusion model to obtain a feature image;

[0112] The text conversion unit is used to convert the feature image into target text through the semantic learning network contained in the image diffusion model;

[0113] An image diffusion unit is used to perform image diffusion processing on the target text and the feature image through the image diffusion network contained in the image diffusion model to obtain a target image;

[0114] An audio conversion unit is used to convert the target text into target audio through a text-to-speech network included in the image diffusion model;

[0115] A video synthesis unit is used to fuse the target image and the target audio to obtain the first video.

[0116] Furthermore, the network structure of the SegNet network includes multiple network layer groups, each network layer group including at least one convolutional layer and at least one pooling layer, and a skip connection is added between the first convolutional layer and the last pooling layer of each network layer group.

[0117] In one implementation of this application, the video input module may include:

[0118] Image template acquisition unit, used to acquire preset image template images;

[0119] The first processing unit is used to input the first video and the image template image as input data to the encoding / decoding network and the ControlNet network for processing, respectively.

[0120] In another implementation of this application embodiment, the video input module may include:

[0121] The prompt text acquisition unit is used to acquire the prompt text corresponding to the virtual image;

[0122] The second processing unit is used to input the first video and the prompt text as input data to the codec network and the ControlNet network for processing, respectively.

[0123] In another implementation of this application, the virtual avatar is the vehicle's in-vehicle voice assistant avatar; the video input module may include:

[0124] The vehicle parameter acquisition unit is used to acquire vehicle parameters of the vehicle during the driving process;

[0125] The third processing unit is used to input the first video and the vehicle parameters as input data to the encoding / decoding network and the ControlNet network for processing, respectively.

[0126] In one implementation of this application, the virtual avatar output module may include:

[0127] A video output unit is used to output a second video through the decoding layer;

[0128] The distance measurement processing unit is used to obtain the range of the virtual image's edge in each frame of the second video; and to use each pixel in the range as input to the distance measurement algorithm to increase the distance between pixels belonging to the virtual image's edge and pixels not belonging to the virtual image's edge in the range, thereby obtaining the processed second video.

[0129] The virtual avatar determination unit is used to determine the processed second video as the generated virtual avatar.

[0130] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for generating a virtual image as described in any of the above embodiments.

[0131] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute a method for generating a virtual image as described in any of the above embodiments.

[0132] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. For example... Figure 6 As shown, the terminal device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the embodiments of the various virtual character generation methods described above, for example... Figure 1 Steps 101 to 104 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 504 are shown.

[0133] The computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 62 in the terminal device 6.

[0134] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0144] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating a virtual avatar, characterized in that, include: Obtain the original image corresponding to the virtual avatar; A first video corresponding to the original image is generated using a trained image diffusion model. The first video is input to the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing; wherein, the encoding / decoding network sequentially includes an encoding layer, an intermediate layer and a decoding layer, and the ControlNet network sequentially includes the same encoding layer, the same intermediate layer and a zero convolutional layer as the encoding / decoding network, and the output of the zero convolutional layer is connected to the input of the decoding layer; The virtual image generated is output through the decoding layer; The step of generating a first video corresponding to the original image using a trained image diffusion model includes: The original image is processed by semantic segmentation using the SegNet network included in the image diffusion model to obtain a feature image; The feature image is converted into target text using the semantic learning network included in the image diffusion model; The target image is obtained by performing image diffusion processing on the target text and the feature image using the image diffusion network contained in the image diffusion model. The target text is converted into target audio using the text-to-speech network included in the image diffusion model; The target image and the target audio are fused to obtain the first video.

2. The method as described in claim 1, characterized in that, The SegNet network structure includes multiple network layer groups, each of which contains at least one convolutional layer and at least one pooling layer, and a skip connection is added between the first convolutional layer and the last pooling layer in each network layer group.

3. The method as described in claim 1, characterized in that, The step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing includes: Get the preset image template; The first video and the image template are used as input data and are respectively input to the encoding / decoding network and the ControlNet network for processing.

4. The method as described in claim 1, characterized in that, The step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing includes: Obtain the prompt text corresponding to the virtual avatar; The first video and the prompt text are used as input data and are respectively input to the codec network and the ControlNet network for processing.

5. The method as described in claim 1, characterized in that, The virtual avatar is the vehicle's in-vehicle voice assistant avatar; the step of inputting the first video into the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing includes: Obtain vehicle parameters during the vehicle's operation; The first video and the vehicle parameters are used as input data and are respectively input to the encoding / decoding network and the ControlNet network for processing.

6. The method according to any one of claims 1 to 5, characterized in that, The virtual image generated through the decoding layer includes: The second video is output through the decoding layer; For each frame of the second video, the range of the image containing the virtual image's edge is obtained; each pixel within the range is used as input to a distance measurement algorithm for distance measurement processing, so as to increase the distance between pixels within the range that belong to the virtual image's edge and pixels that do not belong to the virtual image's edge, thereby obtaining the processed second video; The processed second video is identified as the generated virtual avatar.

7. A device for generating a virtual image, characterized in that, include: The original image acquisition module is used to acquire the original image corresponding to the virtual image; The video generation module is used to generate a first video corresponding to the original image using a trained image diffusion model; A video input module is used to input the first video to the encoding / decoding network included in the image diffusion model and the ControlNet network used to control the image diffusion model for processing; wherein, the encoding / decoding network sequentially includes an encoding layer, an intermediate layer and a decoding layer, and the ControlNet network sequentially includes the same encoding layer, the same intermediate layer and a zero convolutional layer as the encoding / decoding network, and the output of the zero convolutional layer is connected to the input of the decoding layer; A virtual avatar output module is used to output the generated virtual avatar through the decoding layer; The video generation module includes: The feature extraction unit is used to perform image semantic segmentation processing on the original image through the SegNet network included in the image diffusion model to obtain a feature image; The text conversion unit is used to convert the feature image into target text through the semantic learning network contained in the image diffusion model; An image diffusion unit is used to perform image diffusion processing on the target text and the feature image through the image diffusion network contained in the image diffusion model to obtain a target image; An audio conversion unit is used to convert the target text into target audio through a text-to-speech network included in the image diffusion model; A video synthesis unit is used to fuse the target image and the target audio to obtain the first video.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating virtual images as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating virtual images as described in any one of claims 1 to 6.

Images