Video generation method, storage medium and equipment

By removing the foreground area in the digital human video and filling the background image, the problem of unclear display of the digital human video is solved, and a clearer dynamic area edge display effect is achieved.

CN118945284APending Publication Date: 2024-11-12HANGZHOU QIUGUOJIHUA TECHNOLOGY CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202411427897.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-14
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, when generating digital human videos, the display effect of the digital human head edge is poor, resulting in unclear edges of the foreground area of ​​the dynamic video.

Method used

By obtaining pending image and sound information, removing the foreground area images, filling in blank areas to generate a complete background image, and applying the foreground area images to the background image, processing using Gaussian hybrid model and neural network model to ensure edge display Clear.

Benefits of technology

The display clarity of the edges of the dynamic area in dynamic videos is significantly improved, making the edge areas of the foreground area images in the processed videos clearer.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945284A_ABST
    Figure CN118945284A_ABST
Patent Text Reader

Abstract

The invention provides a video generation method, a storage medium and equipment, which can enable the display effect of the background at the edge of the head of a digital person in a generated digital person business card not to be affected by the complex background in the original photo under the condition that the background of the photo is complex. Comprising the steps that a to-be-processed image and sound information are acquired, and the to-be-processed image comprises a background area image and a foreground area image; removing the foreground region image from the to-be-processed image to obtain a first image comprising a blank region and the background region; according to the background region image, filling a blank region of the first image with a pattern having content consistency and coherence with the background region image to obtain a complete background image; for each frame of image of a dynamic video, a foreground area image of each frame of image is acquired, and the dynamic video is a digital video generated according to the to-be-processed image and the sound information; and fitting the foreground region image of each frame of image with the background image to obtain a processed video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, storage medium and device. Background Art

[0002] With the widespread use of smartphones and social media, many people have begun to choose digital human business cards as a tool for communication. Digital human business cards can not only show personal image, but also convey more information, enhancing the fun and interactivity in interpersonal communication. In the process of generating digital human business cards, after the digital human is generated through photos, the digital human is also driven according to the sound to generate a digital human video. However, the digital human video generated by the existing technology often has poor display effect on the edge of the digital human head. Summary of the invention

[0003] One advantage of the present application is that it provides a video generation method, a storage medium and a generation system. Compared with the prior art, in the dynamic video generated by the present application, the edges of the dynamic areas are displayed more clearly.

[0004] Based on this, in order to achieve at least one of the above advantages or other advantages and purposes of the present application, the present application provides a video generation method, comprising the following steps: Acquire an image to be processed and sound information, wherein the image to be processed includes a background area image and a foreground area image; Removing the foreground area image of the image to be processed from the image to be processed to obtain a first image including a blank area and the background area; According to the background area image, a pattern having content consistency and coherence with the background area image is filled in the blank area of ​​the first image to obtain a complete background image; For each frame of a dynamic video, a foreground area image of each frame is obtained, wherein the dynamic video is a digitized video generated based on the obtained image to be processed and the sound information; The foreground area image of each frame image is fitted with the background image to obtain a processed video.

[0005] According to an embodiment of the present application, the process of removing the foreground area image of the image to be processed from the image to be processed to obtain the first image including the blank area and the background area is: Using a difference method based on a Gaussian mixture model, the position of the blank area is determined according to the foreground area image of the image to be processed; The foreground area image is removed to obtain the first image.

[0006] According to an embodiment of the present application, the process of filling a blank area of ​​the first image with a pattern having content consistent with the background area image to obtain a complete background image is specifically as follows: According to the background area image, the blank area is processed by a neural network model to obtain a second image, wherein the contour of the second image is the contour of the blank area, and the content of the second image is consistent and coherent with the content of the background area image; The second image is attached to the blank area of ​​the first image to obtain a complete background image.

[0007] According to one embodiment of the present application, the neural network model includes a generation network and a discrimination network, the generation network is used to generate the second image, the discrimination network is used to identify whether the content of the second image is consistent with the content of the background area image, and the discrimination network includes a local discriminator and a global discriminator.

[0008] According to an embodiment of the present application, for the first frame image of the dynamic video, obtaining the foreground area image of each frame image is specifically: A foreground area image of a current frame image is generated according to the foreground area image position of the image to be processed and the posture parameters of the current frame image; the posture parameters are parameters generated according to the sound information and used to characterize the position information of the foreground area image of the current frame image in the current frame image.

[0009] According to an embodiment of the present application, the image to be processed is any one frame of the dynamic video; for each frame of the dynamic video, obtaining the foreground area image of each frame is specifically: A foreground area image of a current frame image is generated according to the foreground area image position of a first frame image in the dynamic video and the posture parameters of the current frame image; the posture parameters are parameters generated according to the sound information and used to characterize the position information of the foreground area image of the current frame image in the current frame image.

[0010] According to an embodiment of the present application, the process of fitting the foreground area image of each frame image with the background image to obtain the processed video is specifically as follows: Masking the edge of the foreground area image of each frame of image; The foreground area image of each frame image after the masking process is matched with the background image to obtain a processed video.

[0011] According to an embodiment of the present application, the edge of the foreground area image in each frame of image is subjected to masking processing, specifically, the edge of the foreground area image in each frame of image is made transparent.

[0012] The present application also provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned video generation method.

[0013] The present application also provides a device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the above-mentioned video generation method.

[0014] The beneficial effects of the present application are as follows: the present application provides a video generation method, which removes the foreground area image from the image to be processed to obtain a first image including a blank area and the background area, and then fills the blank area with the content that is consistent and coherent with the background image based on the background area image, that is, the content of the background area image is supplemented in the blank area to obtain a complete background image, and then generates a dynamic video based on the image to be processed and sound information, obtains the foreground area image of each frame image in the dynamic video, and fits the foreground area image of each frame image to the complete background image to obtain a processed dynamic video. The present application processes the dynamic video frame by frame. For each frame image, the image itself is a static image, and the foreground area image of each frame image is also a static image. Due to the integrity of the background image and the fact that each frame image uses the same complete background image, after the foreground area image of each frame image is fitted to the complete background image, the edge area of ​​the foreground area image of each frame image is clear. The area where the foreground area image of each frame image is located can be understood as the dynamic area in the dynamic video formed later, thereby making the edge area of ​​the foreground area image in the processed dynamic video clear, that is, improving the display clarity of the dynamic area edge of the dynamic video. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of the video generation method provided for this application; Figure 2A specific flow chart of obtaining a first image in the video generation method provided in this application; Figure 3 A specific flow chart of obtaining a complete background image in the video generation method provided in this application; Figure 4 The effect diagram of the video generation method provided by this application; Figure 5 A specific flow chart of a processed video obtained in a video generation method provided in one embodiment of the present application; Figure 6 A structural diagram of a device provided for other embodiments of the present application; Figure 7 A structural diagram of a video generating device provided in another embodiment of the present application.

[0017] Figure numbers: 11, foreground area image; 12, background area image; 13, blank area; 14, first image; 15, background image; 16, foreground area image of the current frame image. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0019] In the existing video generation method, a dynamic video is directly generated based on the acquired image to be processed and sound information. When the background of the acquired image to be processed is very complex, the display effect of the edge of the foreground area in the generated dynamic video will be deteriorated due to the complex background of the image to be processed. That is, the edge of the foreground area in the dynamic video will be unclear. For example, in the process of making a digital human business card, the user inputs a static image and a sound information, and then generates a digital human dynamic video. The head or hand of the digital human in the dynamic video will match the corresponding action according to the sound information, but the edge of the head or hand of the generated dynamic video will appear unclear.

[0020] In order to solve this problem, the present application provides a video generation method, which can generate a dynamic video with a much higher degree of clarity of the edge display of the dynamic area of ​​the dynamic video after the user inputs static image and sound information than the existing technology.

[0021] Specifically, an embodiment of the present application provides a video generation method, such as Figure 1As shown, it includes the following steps: Step S1, obtaining image and sound information to be processed.

[0022] like Figure 4 As shown, the image to be processed may be a digitized image generated according to an initial image input by a user, or may be an initial image input by a user, wherein the initial image includes but is not limited to an image input by a user (such as a photo), an image stored in a database (animation, cartoon, etc.), etc. The image to be processed is the first frame of the dynamic video.

[0023] In particular, the initial image includes but is not limited to a picture including a portrait, an animal, or a landscape; the sound information includes but is not limited to audio of a person speaking, audio of an animal calling / roaring, natural sounds such as wind and rain, etc. In the embodiment of the present application, a picture containing only a portrait is taken as an example.

[0024] It is understandable that any photo or image can be divided into a background area and a foreground area. Figure 4 As shown, the image to be processed includes a background area image 12 and a foreground area image 11. The background area image 12 and the foreground area image 11 can be distinguished according to the type of the image to be processed, or according to the focus area of ​​the image to be processed, or according to the user's selection. The foreground area image can be understood as the dynamic area of ​​the subsequently generated dynamic video, and the background area image 12 is the static area in the dynamic video. In this embodiment, the division method of the foreground area image 11 and the background area image 12 is not limited.

[0025] Specifically, the background area image 12 and the foreground area image 11 can be determined according to the type of the image to be processed. For example, if the type of the image to be processed is a portrait, the foreground area image 11 is the portrait area of ​​the image to be processed, and the background area image 12 is the non-portrait part of the image to be processed, such as the landscape area behind the portrait. For images containing animals, the foreground area can be the area where the animals are located, and the background area can be the landscape area behind the animals. For images containing plants or natural attractions (such as waterfalls, flowing water, flowers, plants and trees), the foreground area can be the dynamic part of the plants or attractions in the dynamic video, and the background area can be other static parts.

[0026] In other embodiments, the background area image 12 and the foreground area image 11 can be determined according to the focus area of ​​the image to be processed, the focus area of ​​the image to be processed is the foreground area image 11, and the non-focus area of ​​the image to be processed is the background area image 12. The focus area refers to the part focused by the lens during the shooting process, which is usually a clear and detailed area in the picture.

[0027] Furthermore, in other embodiments, the foreground area can be determined according to user needs, that is, one or more foreground areas can be set in an image to be processed according to the user's choice. For example, in an image to be processed containing portraits, animals, raindrops, plants, and buildings, the user can choose to use the portraits, animals, and raindrops as the foreground areas, and the plants and buildings as the background areas. In the dynamic video finally generated, the portraits, animals, and raindrops will move according to the changes in the sound information, while the plants and buildings remain still.

[0028] Step S2: Figure 4 As shown, the foreground area image 11 of the image to be processed is removed from the image to be processed, and a first image 14 including a blank area 13 and the background area is obtained.

[0029] It is worth noting that Figure 4 As shown, the position of the blank area 13 in the first image 14 is consistent with the position of the foreground area image 11 of the image to be processed in the image to be processed. The content of the blank area 13 is empty, that is, transparent.

[0030] Alternatively, if Figure 2 and Figure 4 As shown, the foreground area image 11 is removed from the image to be processed to obtain the first image 14 including the blank area 13 and the background area. The method may be: Step S21, using a difference method based on a Gaussian mixture model to determine the position of the blank area 13 according to the foreground area image 11 of the image to be processed; Step S22 , removing the foreground area image 11 to obtain the first image 14 .

[0031] Optionally, the difference method based on Gaussian mixture model is a MOG differentiator, and the specific use of the MOG differentiator is as follows: 1. Model initialization: Initialize the Gaussian mixture model of each pixel. Usually, several Gaussian distributions are used to fit the color value of the pixel in the image. 2. Model update: Update the Gaussian model based on the pixel values ​​of the current image. This process involves comparing each pixel value with the various Gaussian distributions of the current model to determine which Gaussian distribution it belongs to; 3. Foreground and background separation: For each pixel, whether it belongs to the foreground is usually determined based on the weight and distribution of the Gaussian distribution (for example, if a pixel value does not match the Gaussian distribution in the background model, the pixel is considered to be the foreground); 4. Post-processing,Finally, some post-processing operations (such as morphological,operations) are usually performed to remove noise or fill holes,to obtain more accurate foreground detection results.

[0032] In addition, in other embodiments, a cutout method may be adopted to remove the foreground area image 11. The present application does not limit the specific method of removing the foreground area image in this process.

[0033] Step S3 , according to the background area image 12 , filling the blank area 13 of the first image 14 with a pattern that has content consistency and coherence with the background area image 12 to obtain a complete background image 15 .

[0034] That is to say, according to the content and logic of the background area image, the blank area is filled to obtain a complete background image consisting entirely of the background area image. For example, the content of the image to be processed includes people, plants, raindrops and animals.

[0035] If the user selects a person as the foreground area image 11 , the person area is removed to generate content that is consistent with the content of the non-person area in the image to be processed, thereby obtaining a complete background image 15 .

[0036] If the user selects people and plants as the foreground area image 11, the person area and the plant area are removed to generate content that is consistent with the content of the non-person and non-plant areas in the image to be processed, thereby obtaining a complete background image 15.

[0037] It should be noted that the method for filling the blank area 13 is to directly generate a pattern with content consistency and coherence with the background area image in the blank area 13 of the first image to directly obtain a complete background image, or to first generate an independent image with the same outline as the blank area and then fit the image into the first image. This embodiment is described using the latter as an example.

[0038] It is worth noting that Figure 3 and Figure 4 As shown, the method of filling the blank area 13 in step S3 is specifically as follows: Step S31, according to the background area image 12, the blank area 13 is processed by a neural network model to obtain a second image, the outline of the second image is the outline of the blank area 13, and the content of the second image is consistent and coherent with the content of the background area image 12.

[0039] For example, the contents of an image to be processed include people, plants, raindrops and animals.

[0040] If the user selects a person as the foreground area image 11, the blank area 13 is processed through a neural network model to generate a second image having content consistent with the content of the non-person area in the image to be processed and having the same outline as the blank area outline.

[0041] If the user selects people and plants as the foreground area image 11, the blank area 13 is processed through the neural network model to generate a second image whose content is consistent with the content of the non-people and non-plant areas in the image to be processed and whose contour is the same as the contour of the blank area 13.

[0042] Step S32 , attaching the second image to the blank area 13 of the first image 14 to obtain a complete background image 15 .

[0043] It is worth noting that the neural network model includes a generation network and a discrimination network, wherein the generation network is used to generate the second image, and the discrimination network is used to discriminate whether the content of the second image is consistent with the content of the background area image 12. The discrimination network includes a local discriminator and a global discriminator.

[0044] Optionally, the neural network model includes a generating network and a discriminating network. The neural network model as a whole applies a ReLU activation function to increase nonlinear capabilities and help activate useful features. The process of the neural network model outputting the second image uses a Sigmoid activation function to ensure that the output pixel value is between 0 and 1, so that the neural network model can directly output the pixel value of the image, that is, the final result is output as the second image in the form of pixel values. The generating network uses a convolutional neural network (CNN) as a basic architecture (CNN is good at capturing spatial structural information in images) to generate relevant data of the second image. The discriminating network includes a local discriminator and a global discriminator. The local discriminator is used to check the authenticity of a specific area in the image, that is, the authenticity of the second image. The global discriminator is used to determine whether the entire image is authentic, that is, to fit the first image 14 with the second image, and to determine whether the image after the first image 14 and the second image are fitted is authentic.

[0045] It should be noted that the neural network model in this embodiment may be a general neural network model, or a neural network model trained with images of a specific category, such as a neural network model specially trained for plant images.

[0046] Step S4: for each frame of the dynamic video, obtain the foreground area image of each frame of the dynamic video, wherein the dynamic video is a digitized video generated based on the obtained image to be processed and the sound information.

[0047] It is worth noting that obtaining the foreground area image of each frame image can be understood as the foreground area image 16 of each current frame image.

[0048] It should be noted that, for each current frame image, the foreground area image 16 can be obtained by comprehensively acquiring the foreground area image 11 of the reference image and the posture parameters of the current frame image. The reference image can be the first frame image of the dynamic video, or it can be an initial image input by the user. The initial image can be a digitized image (such as a cartoon image) or an initial photo.

[0049] In this embodiment, the reference image is the first frame image of the dynamic video.

[0050] When the reference image is the initial image input by the user, for each frame image of the dynamic video, the step of obtaining the foreground area image 11 of each frame image is specifically as follows: according to the position of the foreground area image 11 of the image to be processed and the posture parameters of the current frame image, the foreground area image 16 of the current frame image is generated; the posture parameters are parameters generated according to the sound information, and are used to characterize the position information of the foreground area image 16 of the current frame image in the current frame image.

[0051] It is understandable that, for the generated dynamic video, since it is a dynamic video generated based on sound information, and the initial image input by the user is a static image, for different sound information, the generated dynamic video may not necessarily include the action of the foreground area image 11 of the initial image. If the initial image input by the user is a portrait with hands drooping, and there is no action of the portrait with hands drooping in the dynamic video, the initial image can also be used as a reference image.

[0052] In other embodiments, when the reference image is any frame image of a dynamic video, for each frame image of the dynamic video, the step of obtaining the foreground area image 16 of each current frame image is specifically as follows: According to the position of the foreground area image 11 of the first frame image in the dynamic video and the posture parameters of the current frame image, the foreground area image 16 of the current frame image is generated; the posture parameters are generated according to the sound information and are used to characterize the position information of the foreground area image 16 of the current frame image in the current frame image.

[0053] Specifically, for example, the content of an image to be processed includes people, plants, raindrops and animals.

[0054] If the user selects the character part as the foreground area image 11, then the process of generating the foreground area image 16 of the current frame image is to generate the foreground area image 16 of the current frame image according to the character part of the first frame image in the dynamic video and the posture parameters of the current frame image.

[0055] If the user selects human and plant parts as the foreground area image 11, then the process of generating the foreground area image 16 of the current frame image is to generate the foreground area image 16 of the current frame image based on the human and plant parts of the first frame image in the dynamic video and the posture parameters of the current frame image.

[0056] In particular, in the process of generating the posture parameters, the posture parameters that conform to the characteristics of the character are generated according to the style of the sound information.

[0057] In particular, the posture parameters are generated using a variational autoencoder (VAE) based on the foreground area image 11 in the dynamic video and the sound information. Specifically: in the process of generating the posture parameters, the variational autoencoder is based on the encoder-decoder framework and uses the VAE method to construct the posture parameters. VAE Encoder and VAE Decoder are both two-layer MLP (multi-layer perceptron) networks. The VAE Encoder is an encoder, and its task is to extract features from the input data and encode the features into a latent variable vector Z, which will be embedded in a Gaussian distribution. Specifically, for the head posture coefficient ρ0, the VAEencoder will output two vectors: the mean μ and the variance σ 2 , then from N(μ,σ 2 ) to obtain the latent variable vector Z. The VAE Decoder is a decoder for recovering data from the latent space. Specifically, the VAE Decoder will reconstruct the head posture of consecutive t frames based on the latent variable vector Z sampled from the Gaussian distribution. In addition, the VAE Decoder accepts the character style feature Z_style and the audio clip As a conditional input, the generated animated head pose not only conforms to the character's style, but also can be synchronized with the sound to show rhythmic movements.

[0058] Step S5, the foreground area image of each frame image is fitted with the background image 15 to obtain a processed video.

[0059] It is worth noting that the foreground area image 11 is removed from the image to be processed to obtain a first image 14 including a blank area 13 and the background area, and then the blank area 13 is filled with the content consistent and coherent with the background image 15 according to the background area image 12, that is, the blank area 13 is supplemented with the content of the background area image 12 to obtain a complete background image 15, and then a dynamic video is generated according to the image to be processed and the sound information, and the foreground area image 16 of each current frame image in the dynamic video is obtained, and the foreground area image 16 of each current frame image is pasted to the complete background image 15 to obtain a processed dynamic video.

[0060] The present application processes the dynamic video frame by frame. For each frame image, the image itself is a static image, and the foreground area image 16 of each current frame image is also a static image. Due to the integrity of the background image 15 and the fact that each frame image uses the same complete background image 15, after the foreground area image 16 of each current frame image is attached to the complete background image 15, the edge area of ​​the foreground area image 16 of each current frame image is clear. The area where the foreground area image 11 is located can be understood as the dynamic area in the dynamic video formed later, thereby making the edge area of ​​the foreground area image 11 in the processed dynamic video clear, that is, the display clarity of the dynamic area edge of the dynamic video is improved.

[0061] In other embodiments, in the process of fitting the foreground area image of each frame image with the background image 15 to obtain the processed video, in order to further optimize the clarity of the edge of the dynamic area, the edge of the foreground area image may be blurred before fitting, such as Figure 5 As shown, specifically: Step S51, masking the edge of the foreground area image 16 of each current frame image, that is, blurring or making it transparent; Step S52, the foreground area image 16 of the current frame image after the masking process is fitted with the background image 15 to obtain a processed video.

[0062] In particular, the step S51 is specifically to make the edge of the foreground area image 16 of each current frame image transparent.

[0063] Another embodiment of the present application also provides a video generation method. The embodiment shown is different from the previous embodiment in that the image to be processed can be any frame of the dynamic video, and the process of step S5 uses image synthesis technology to make the edge of the foreground area image 16 of the current frame image transparent, and then the foreground area image 16 of the current frame image after transparency is superimposed on the background image 15 to obtain a processed video.

[0064] It should be noted that, in another embodiment of the present application, the image synthesis technique used is Alpha blending, which is used to blend the foreground region image 16 of the current frame image with transparency information with the background image 15. The transparency information is usually stored in the Alpha channel of an image, and this channel defines the opacity level of each pixel.

[0065] The basic formula for Alpha blending is as follows: output = foreground × mask + background × (1 - mask) Where output refers to the output result, that is, any one frame image in the processed video; foreground is the foreground region image 16 of the current frame image; background is the background image; mask refers to the value of the Alpha channel, which is usually a floating-point number between 0 and 1. In actual calculation, if the Alpha channel is stored with 8-bit precision (such as the common PNG format), then the original value range of mask is [0, 255], but it will be normalized to the range of [0, 1] during calculation, that is, the original value of mask is divided by 255.

[0066] When the value of mask is 0, the pixel values of the edge part of the foreground region image 16 of the output current frame image belong to the background; when the value of mask is 1, the pixel values of the edge part of the foreground region image 16 of the output current frame image belong to the foreground; when 0 < mask < 1, the pixel values of the edge part of the foreground region image 16 of the output current frame image are a blend of the foreground and the background.

[0067] For example, in this embodiment, the value of mask is 0.5, then the pixel output values of the edge part of the foreground region image 16 of the output current frame image are a blend of the foreground and the background.

[0068] In other embodiments of the present application, a storage medium is provided, and the storage medium includes a stored program. Wherein, when the program runs, it controls the device where the storage medium is located to execute the above video generation method.

[0069] Specifically, the storage medium may include, but is not limited to, a hard disk, an optical disk, a USB flash drive, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a magnetic tape, a floppy disk, and an optical storage device. These storage media may be implemented by any appropriate method or technology, as long as they can store program instructions and these instructions can be run on the device. The program includes a series of instructions, and when these instructions are executed, the device can complete the various steps in the above-mentioned video generation method.

[0070] It is worth noting that the program on the storage medium can be an independent software package or a module integrated into a larger system. It can be deployed on various types of computing devices, including but not limited to personal computers (PCs), servers, mobile devices (such as smartphones and tablets), dedicated hardware devices, etc.

[0071] In particular, the computer-readable storage medium can be widely used in multiple fields, such as digital entertainment, online education, e-commerce, virtual reality (VR) and augmented reality (AR), etc. Applications in these fields require high-quality video generation technology.

[0072] In other embodiments of the present application, a device is also provided, such as Figure 6 As shown, it includes: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include programs for executing the above-mentioned video generation method.

[0073] In particular, the processor is one or more central processing units (CPUs) or graphics processing units (GPUs) for performing computing tasks. These processors can be single-core or multi-core and can be integrated on one or more chips. In high-performance computing scenarios, GPUs are particularly suitable for parallel computing tasks that process large amounts of image data.

[0074] In particular, the types of memory include volatile memory (such as RAM) and non-volatile memory (such as hard disk drives, solid-state drives, flash memory, etc.). The memory is used to store program code, intermediate calculation results, and final output data. Among them, volatile memory is mainly used to temporarily store data being processed, while non-volatile memory is used for persistent storage.

[0075] In particular, the one or more programs are stored in a memory and are configured to be executed by one or more processors. These programs contain instructions required to implement the above-mentioned video generation method, including but not limited to functional modules such as image acquisition, preprocessing, feature extraction, model application, result generation, post-processing and result output.

[0076] It is worth noting that Figure 7 As shown, another embodiment of the present application further discloses a video generating device, including: A storage module, used for storing the image to be processed, the sound information, the first image, the background image and the dynamic video; An acquisition module, used for acquiring an image to be processed and sound information, wherein the image to be processed includes a background area image and a foreground area image, and the acquisition module may be an input interface of the electronic device; A removal module, used for removing the foreground area image from the image to be processed to obtain a first image including a blank area and the background area; A filling module, configured to fill, according to the background area image, a pattern having content consistency and coherence with the background area image in a blank area of ​​the first image, so as to obtain a complete background image; A generating module, for acquiring a foreground area image of each frame of a dynamic video, wherein the dynamic video is a digitized video generated based on the acquired image to be processed and the sound information; The fusion module is used to fit the foreground area image of each current frame image with the background image to obtain a processed video.

[0077] It is worth noting that the storage module, the acquisition module, the removal module, the filling module, the generation module and the fusion module cooperate together to implement the above-mentioned video generation method.

[0078] Beneficial effect: The storage module is simultaneously connected to the acquisition module, the removal module, the filling module, the generation module and the fusion module to input / output the data such as the image to be processed, the sound information, the first image, the background image and the dynamic video stored in the storage module, laying a storage foundation for the video generation device to implement the above-mentioned video generation method. The acquisition module acquires the image to be processed and the sound information, providing a data basis for the video generation device to implement the above-mentioned video generation method. The removal module and the filling module cooperate to obtain a complete background image. The generation module generates a foreground area image of each frame image (that is, the foreground area image 16 of each current frame image). The fusion module fits the foreground area image 16 of each current frame image with the background image respectively, and finally obtains the processed video, that is, the above-mentioned video generation method is implemented.

[0079] In some possible implementations, various aspects of the video generation method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the video generation method according to various exemplary implementations of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 1 Follow the steps shown in .

[0080] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0081] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0082] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0083] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0084] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user's computer device, partially on the user's computer device, as a separate software package, partially on the user's computer device and partially on a remote computer device, or completely on the remote computer device. In the case of a remote computer device, the remote computer device can be connected to the user's computer device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer device (for example, using an Internet service provider to connect through the Internet).

[0085] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0086] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0087] The video generation method, the computer-readable storage medium, the device, and the video generation apparatus can be widely used in multiple fields such as virtual reality (VR) and augmented reality (AR), remote communication, digital entertainment, education and training, e-commerce, and social applications.

[0088] In a virtual reality environment, clear digital humans can provide a more realistic experience. The application range of this technology is very wide, covering a variety of fields from entertainment to education. In the field of entertainment, digital human technology can create lifelike characters to immerse players in the virtual world. For example, in multiplayer online games, clear digital humans can enhance the interactive experience between players and make the game more realistic and engaging. In addition, in virtual concerts or sports events, digital humans can appear as performers or athletes, bringing a new visual experience to the audience. In the field of training and simulation: In professional training and simulation scenarios, digital humans can be used to simulate real working environments to help trainees better master the required skills. For example, in medical training, digital humans can simulate patients to allow medical students to practice diagnostic and treatment skills; in military training, digital humans can simulate enemy forces to help soldiers conduct tactical drills. In the field of cultural tourism, digital humans can serve as tour guides to lead tourists to visit historical sites or virtual attractions, providing rich historical background and cultural interpretation, making the tourism experience richer and more interesting.

[0089] Augmented reality (AR) combines virtual elements with the real world to provide users with a new interactive experience. Clear digital humans can greatly enhance the immersion and practicality of AR applications. Education: Digital humans can serve as virtual teachers or classmates to help students better understand abstract concepts. For example, in scientific experiments, digital humans can demonstrate complex physical phenomena, allowing students to intuitively see the results of experiments. Commercial display: In the retail industry, digital humans can be used to display product features, allowing consumers to experience the effects of using goods before shopping. For example, in a furniture store, customers can use AR technology to see the actual effects of furniture placed in their homes. Healthcare: In the field of healthcare, digital humans can be used to simulate human organs to help doctors plan surgeries or educate patients. For example, before surgery, doctors can use AR technology to observe the location of lesions simulated by digital humans and develop more accurate surgical plans.

[0090] In the field of remote communication: In video conferences, clear digital humans can improve the quality of communication. Especially within multinational companies, clear digital humans can enhance the collaborative effect of remote work. Business meetings: Within the enterprise, clear digital humans can be used for remote meetings, allowing colleagues in different regions to communicate without barriers. For example, in a project discussion meeting, a digital human can show the project progress in real time and interact with other team members. Customer service: In the field of customer service, digital humans can serve as virtual customer service representatives to provide 24-hour uninterrupted service. For example, in banks or telecommunications companies, digital humans can answer common customer questions and improve service efficiency. Distance education: In distance education, clear digital humans can serve as teacher assistants to assist online teaching. For example, in an online class, a digital human can demonstrate experimental steps to help students better understand knowledge points.

[0091] In the field of digital entertainment: In filmmaking, clear digital humans can create a more immersive effect. With the development of computer graphics, digital human technology has become an indispensable part of modern filmmaking. Special effects production: In science fiction or fantasy movies, digital humans can be used to create fictional characters or creatures to bring shocking visual effects to the audience. For example, in "Avatar", digital human technology was used to create the Na'vi people, showing an amazing other world. Stand-in actors: In dangerous or difficult scenes, digital humans can replace real actors to ensure safe shooting. For example, in high-altitude jumps or high-speed chase scenes, digital humans can simulate the actions of actors to avoid accidents. In video games, clear digital humans can enhance the immersion of the game and make it easier for players to invest in the storyline. Character design: In role-playing games, clear digital humans can be used to design unique game characters, giving players more choices. In many current games, players can freely customize the appearance and personality characteristics of the protagonist. Interactive plot: In narrative games, clear digital humans can enhance the appeal of the story and make players more involved. In many current games, digital human technology is used to shape multiple main characters, allowing players to experience complex emotional conflicts.

[0092] In the field of education and training: In online education, clear digital humans can be used to demonstrate teaching content and increase students' interest in learning. Experimental demonstration: In science courses, clear digital humans can demonstrate complex experimental processes to help students understand abstract concepts. For example, in a chemistry experiment class, digital humans can show the microscopic changes of chemical reactions, allowing students to intuitively see the changes at the molecular level. Scenario simulation: In language learning, clear digital humans can simulate real conversation scenarios to help students practice oral expression. For example, in an English listening class, digital humans can conduct conversation exercises with students to improve their listening skills.

[0093] In the field of product display: In e-commerce platforms, clear digital people can show the trial effect of products and increase consumers' willingness to buy. Clothing try-on: In clothing sales, clear digital people can simulate the try-on effect, allowing consumers to see the actual wearing effect of clothes before buying. For example, on Taobao or JD.com, digital people can show the matching effect of clothes of different styles and colors. Home decoration: In the sale of home furnishings, clear digital people can show the effect of furniture placement, helping consumers to better plan the layout of space. For example, in IKEA, digital people can show the placement effect of furniture such as sofas and coffee tables, allowing consumers to see what the room will look like after the arrangement in advance.

[0094] In the field of social applications: In social networks, clear digital people can serve as the user's virtual image to enhance the user's social experience. Personal homepage: On social media platforms, clear digital people can serve as the user's virtual image to show the user's personality and interests. For example, in WeChat Moments or Douyin, digital people can post updates or share videos on behalf of users. Virtual gatherings: In virtual gathering applications, clear digital people can allow users to participate in online activities in a more personalized way. For example, on Tencent Conference or Zoom, digital people can participate in online meetings or parties on behalf of users to enhance the fun of interaction.

[0095] It should be noted that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment.

[0096] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first feature is directly in contact with the second feature, or the first feature and the second feature are indirectly in contact through an intermediate medium. Moreover, a first feature being “above”, “above” or “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below” or “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0097] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more related listed items.

[0098] The technical features of the above embodiments can be combined without changing the basic principles of the present application. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of patent protection of the present application shall be subject to the attached claims.

Claims

1. A video generation method, characterized in that: The following steps are involved: Acquire an image to be processed and sound information, wherein the image to be processed includes a background area image and a foreground area image; Removing the foreground area image of the image to be processed from the image to be processed to obtain a first image including a blank area and the background area; According to the background area image, a pattern having content consistency and coherence with the background area image is filled in the blank area of ​​the first image to obtain a complete background image; For each frame of a dynamic video, a foreground area image of each frame is obtained, wherein the dynamic video is a digitized video generated based on the obtained image to be processed and the sound information; The foreground area image of each frame image is fitted with the background image to obtain a processed video.

2. The video generation method according to claim 1, characterized in that: The process of removing the foreground area image of the image to be processed from the image to be processed to obtain the first image including the blank area and the background area is: Using a difference method based on a Gaussian mixture model, the position of the blank area is determined according to the foreground area image of the image to be processed; The foreground area image is removed to obtain the first image.

3. The video generation method according to claim 1, characterized in that: The process of filling the blank area of ​​the first image with a pattern having content consistent with the background area image according to the background area image to obtain a complete background image is specifically as follows: According to the background area image, the blank area is processed by a neural network model to obtain a second image, wherein the contour of the second image is the contour of the blank area, and the content of the second image is consistent and coherent with the content of the background area image; The second image is attached to the blank area of ​​the first image to obtain a complete background image.

4. The video generation method according to claim 3, characterized in that: The neural network model includes a generation network and a discrimination network, the generation network is used to generate the second image, the discrimination network is used to identify whether the content of the second image is consistent with the content of the background area image, and the discrimination network includes a local discriminator and a global discriminator.

5. The video generation method according to claim 1, characterized in that: For the first frame image of the dynamic video, obtaining the foreground area image of each frame image is specifically as follows: A foreground area image of the current frame image is generated according to the foreground area image position of the image to be processed and the posture parameters of the current frame image; the posture parameters are parameters generated according to the sound information and used to characterize the position information of the foreground area image of the current frame image in the current frame image.

6. The video generation method according to claim 1, characterized in that: The image to be processed is any one frame of the dynamic video; for each frame of the dynamic video, obtaining the foreground area image of each frame is specifically: A foreground area image of a current frame image is generated based on the foreground area image position of the first frame image in the dynamic video and the posture parameters of the current frame image; the posture parameters are parameters generated based on the sound information and used to characterize the position information of the foreground area image of the current frame image in the current frame image.

7. The video generation method according to claim 1, characterized in that: The process of fitting the foreground area image of each frame of image with the background image to obtain the processed video is specifically as follows: Masking the edge of the foreground area image of each frame of image; The foreground area image of each frame image after the masking process is matched with the background image to obtain a processed video.

8. The video generation method according to claim 7, characterized in that: The masking process is performed on the edge of the foreground area image of each frame image, specifically, the edge of the foreground area image in each frame image is made transparent.

9. A storage medium, characterized in that: The storage medium includes a stored program, Wherein, when the program is running, the device where the storage medium is located is controlled to execute any of claims 1 to 8. A video generation method as described above.

10. A device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the video generation method described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Video data processing method and device and computing device

    CN107633228A

  • Image restoration method based on generative adversarial neural network

    CN111292265A

  • Large-scale panoramic viewpoint synthesis method based on single-viewpoint RGB-D image

    CN111325693A

  • Video generation method and device, equipment and storage medium

    CN114040129A

  • Picture-to-video conversion method and device, equipment and storage medium

    CN114331828A