Video generation method and apparatus, storage medium and electronic device

Abstract

Disclosed in the present application are a video generation method and apparatus, a storage medium and an electronic device. The method comprises: acquiring a content description text and a content reference video, the content description text comprising information for describing target content expressed by a target video expected to be generated, and the content reference video comprising action reference information related to the target content; performing feature extraction on the content description text to obtain text semantic features, the text semantic features being used for representing semantic information of the content description text; performing feature extraction on the content reference video to obtain video reference features, the video reference features being used for representing action reference information in the content reference video; and generating a target video on the basis of the text semantic features and the video reference features. The present application can improve the quality of generated target videos.

Description

Video generation method, device, storage medium, and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 26, 2023, with application number 2023109234935 and application name “Video Acquisition Method, Device, Storage Medium and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and more specifically to the field of computer vision technology. Background Art

[0003] In video generation scenarios, artificial intelligence (AI) models are usually used to generate a series of images based on the input content description text, and then use this series of images to compose a coherent video.

[0004] It can be seen that the content description text in the above method is specifically used to generate images, rather than directly used to generate videos. Since the generation process of each image is independent of each other, it is easy for the generated images to have poor coherence. Accordingly, in the video composed of various images, obvious jitter may occur between consecutive frames, affecting the quality of the generated video, that is, the quality of the generated video is poor.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a video generation method, device, storage medium, and electronic device, which can improve the quality of the generated video.

[0007] According to one aspect of an embodiment of the present application, a video generation method is provided, which is performed by an electronic device and includes:

[0008] Obtaining a content description text and a content reference video, wherein the content description text includes information describing the target content expressed by the target video to be generated, and the content reference video includes action reference information related to the target content;

[0009] Performing feature extraction on the content description text to obtain text semantic features, wherein the text semantic features are used to represent semantic information of the content description text; performing feature extraction on the content reference video to obtain video reference features, wherein the video reference features are used to represent the action reference information in the content reference video;

[0010] Generate a target video based on the above text semantic features and the above video reference features.

[0011] According to another aspect of the embodiments of the present application, a video generating apparatus is provided, including:

[0012] A first acquisition unit is configured to acquire a content description text and a content reference video, wherein the content description text includes information describing a target content expressed by a target video to be generated, and the content reference video includes action reference information related to the target content;

[0013] an extraction unit configured to perform feature extraction on the content description text to obtain text semantic features, wherein the text semantic features are used to represent semantic information of the content description text; and to perform feature extraction on the content reference video to obtain video reference features, wherein the video reference features are used to represent the action reference information in the content reference video;

[0014] The generating unit is used to generate a target video according to the above text semantic features and the above video reference features.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, which includes a stored program, wherein the program executes the above video generation method when executed by an electronic device.

[0016] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described video generation method.

[0017] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the video generation method through the computer program.

[0018] In an embodiment of the present application, a content description text and a content reference video are obtained, wherein the content description text includes information describing the target content expressed by the target video to be generated, and the content reference video includes action reference information related to the target content; feature extraction is performed on the content description text to obtain text semantic features, wherein the text semantic features are used to characterize the semantic information of the content description text; feature extraction is performed on the content reference video to obtain video reference features, wherein the video reference features are used to characterize the action reference information in the content reference video; and a target video is generated based on the text semantic features and the video reference features. In an embodiment of the present application, the target content expressed by the target video to be generated is described by the content description text, and a content reference video with video coherence characteristics is used to provide a reference for the target content. In this way, the action information in the coherent content reference video can be fully referenced during the generation of the target video, thereby making the actions in the generated video content more coherent, and making the generated target video more coherent, thereby achieving the purpose of obtaining a higher quality output video, thereby achieving the technical effect of improving the accuracy of video acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] FIG1 is a schematic diagram of an application environment of an optional video generation method according to an embodiment of the present application;

[0021] FIG2 is a schematic diagram of a process of an optional video generation method according to an embodiment of the present application;

[0022] FIG3 is a schematic diagram of an optional video generation method according to an embodiment of the present application;

[0023] FIG4 is a schematic diagram of another optional video generation method according to an embodiment of the present application;

[0024] FIG5 is a schematic diagram of another optional video generation method according to an embodiment of the present application;

[0025] FIG6 is a schematic diagram of another optional video generation method according to an embodiment of the present application;

[0026] FIG7 is a schematic diagram of another optional video generation method according to an embodiment of the present application;

[0027] FIG8 is a schematic diagram of another optional video generation method according to an embodiment of the present application;

[0028] FIG9 is a schematic diagram of another optional video generation method according to an embodiment of the present application;

[0029] FIG10 is a schematic diagram of an optional video generating device according to an embodiment of the present application;

[0030] FIG11 is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] According to one aspect of an embodiment of the present application, a video generation method is provided. Optionally, as an optional implementation, the video generation method may be, but is not limited to, applied in the environment shown in FIG1 . The environment may include, but is not limited to, a user device 102 and a server 112 . The user device 102 may include a display 104 , a processor 106 , and a memory 108 . The server 112 may include a database 114 and a processing engine 116 .

[0034] The specific process can be as follows:

[0035] Step S102: the user device 102 obtains a content description text and a content reference video;

[0036] Steps S104-S106: User device 102 sends the content description text and content reference video to server 112 via network 110;

[0037] In steps S108-S112, the server 112 uses the processing engine 116 to extract features from the content description text to obtain text semantic features, and extract features from the content reference video to obtain video reference features; and further generates a target video based on the text semantic features and the video reference features.

[0038] In steps S114 - S116 , the server 112 sends the target video to the user device 102 via the network 110 . The user device 102 displays the target video on the display 104 via the processor 106 and stores the target video in the memory 108 .

[0039] In addition to the example shown in FIG1 , the above steps may also be performed independently by a user device or a server, or by a user device and a server working together. For example, the user device 102 may perform steps S108-S112, thereby reducing the processing pressure on the server 112. The user device 102 includes, but is not limited to, a handheld device (such as a mobile phone), a laptop computer, a tablet computer, a desktop computer, a vehicle-mounted device, a smart TV, etc., and this application does not limit the specific form of the user device 102. The server 112 may be a single server, a server cluster consisting of multiple servers, or a cloud server.

[0040] Optionally, as an optional implementation, as shown in FIG2 , the video generation method may be executed by an electronic device, such as the user device or server shown in FIG1 , and the specific steps include:

[0041] S202, obtaining a content description text and a content reference video, wherein the content description text includes information describing the target content expressed by the target video to be generated, and the content reference video includes action reference information related to the target content;

[0042] S204: extracting features from the content description text to obtain text semantic features, wherein the text semantic features are used to represent semantic information of the content description text; extracting features from the content reference video to obtain video reference features, wherein the video reference features are used to represent the action reference information in the content reference video;

[0043] S206: Generate a target video based on the text semantic features and the video reference features.

[0044] Optionally, in this embodiment, the above-mentioned video generation method can be applied to, but is not limited to, generative artificial intelligence (AIGC) application scenarios. AIGC refers to artificial intelligence technology capable of generating new content, audio, images, and the like, such as AI-generated images and AI-generated videos. This embodiment combines content description text with content reference videos, allowing users to input content reference videos and generate video content based on the content description text. This effectively improves the controllability and quality of the generated video content.

[0045] Optionally, in this embodiment, the content description text describing the target content expressed by the target video to be generated may involve information on objects, scenes, actions, emotions, etc. in the video.

[0046] To give further examples, for example, the content description text can be used to provide a general description of the entire video, including basic information such as the theme, scene, time and place of the video, for example, "A red car drives by in the picture"; or, the content description text can be used to describe the actions or behaviors of objects (such as people or objects) in the video, for example, "A puppy chases a ball"; or, the content description text can be used to evaluate the emotional color of the video content, for example, "It shows a warm family moment"; or, the content description text can be used to describe the scene or environment presented in the video, for example, "The video shows a beautiful beach"; or, the content description text can be used to describe the events or changes that occur in the video in chronological order, for example, "At the beginning, the sun rises slowly, and then ushers in a spectacular sunset."

[0047] Optionally, in this embodiment, in order to improve the relevance between the input content description text and the output target video, during the video generation process, a content reference video provided as a reference for the target content can be combined. For example, the content description text is "a puppy chasing a ball", and the content reference video shows a series of actions of a kitten chasing a ball. The content description text and the content reference video can then be combined to ultimately generate a target video showing a series of actions of a puppy chasing a ball. The series of actions shown in the target video can be similar to, but not limited to, the series of actions shown in the content reference video.

[0048] Optionally, in this embodiment, text semantic features are used to characterize the semantic information of the content description text, that is, the semantic information of the content description text describing the target content, which can be but is not limited to the expression of the meaning and information carried by the content description text, such as the vocabulary selection, word meaning and part of speech in the text reflecting the meaning and connotation of the text, the sentence structure, grammatical rules and the relationship between words in the text, which can reflect the logical and semantic connection between sentences, the contextual environment in the text, the tone and emotional color in the text, the themes and topics involved in the text and related professional field knowledge, etc.

[0049] Optionally, in this embodiment, the video reference feature is used to represent the key information that the content reference video provides as a reference for the target content, specifically, the action reference information within the content reference video. To improve the coherence of the target video, the video reference feature may, but is not limited to, be a feature that meets dynamic characteristics. Furthermore, the key information may, but is not limited to, correspond to key content within the target content, where the key content may, but is not limited to, be dynamic content.

[0050] To further illustrate, optionally, as shown in FIG3 , a target video 308 for showing a puppy dancing is generated by combining a content reference video 302 and a content description text 304 , wherein the video reference feature 306 obtained by extracting features from the content reference video 302 may be, but is not limited to, a dance movement feature in the content reference video 302 . The reason why the dance movement feature in the content reference video 302 is extracted as the video reference feature 306 is, firstly, because the dance movement feature meets the dynamic characteristic condition, and secondly, because the “dancing” in the content description text 304 “a puppy dancing” also belongs to dynamic content, and “dancing” corresponds to the dance movement feature. Therefore, the dance movement feature in the content reference video 302 is extracted as the video reference feature 306 .

[0051] Optionally, in this embodiment, a target video is generated based on text semantic features and video reference features, such as the text semantic features provide the desired video theme, content and keywords, and at least one video element is obtained based on the video theme, content and keywords, and then the action reference features provided by the video reference features are used to guide the generation of key video elements in the video elements, thereby obtaining the target video, wherein the text semantic features ensure the consistency between the target video and the desired target content, while the video reference features ensure the video quality of the target video.

[0052] It should be noted that the use of content description text to describe the target content expressed in the desired generated video, and the use of content reference video with video coherence characteristics to provide a reference for the target content, make the generated video content more coherent, thereby improving the correlation between the input content description text and the output target video, thereby achieving the technical effect of improving the accuracy of video generation and being able to generate higher quality videos.

[0053] To further illustrate, optionally, based on the scenario shown in FIG3 , continuing as shown in FIG4 , a content description text 304 and a content reference video 302 are generated, wherein the content description text 304 includes information describing the target content expressed by the desired generated video, and the content reference video 302 includes action reference information providing a reference for the target content; feature extraction is performed on the content description text 304 to obtain text semantic features 402, wherein the text semantic features 402 are used to characterize the semantic information of the content description text 304 describing the target content; feature extraction is performed on the content reference video 302 to obtain video reference features 306, wherein the video reference features 306 are used to characterize the action reference information provided by the content reference video 302 for the target content, such as the key information providing a video generation reference for “dancing” in the target content is the dancing action information in the content reference video 302; and a target video 308 is generated based on the text semantic features 402 and the video reference features 306.

[0054] Through the embodiments provided in this application, a content description text and a content reference video are obtained, wherein the content description text includes information describing the target content expressed by the target video to be generated, and the content reference video includes action reference information related to the target content; feature extraction is performed on the content description text to obtain text semantic features, wherein the text semantic features are used to characterize the semantic information of the content description text; feature extraction is performed on the content reference video to obtain video reference features, wherein the video reference features are used to characterize the action reference information in the content reference video; and the target video is generated using the text semantic features and the video reference features. By using the content description text to describe the target content expressed by the video to be generated, and using the content reference video with video coherence characteristics to provide a reference for the target content, the action information in the coherent content reference video can be fully referenced during the generation of the target video, thereby making the actions in the generated video content more coherent, and also making the generated target video more coherent, thereby achieving the purpose of obtaining a higher quality output video, thereby achieving the technical effect of improving the accuracy of video generation.

[0055] As an optional solution, a target video is generated based on text semantic features and video reference features, including:

[0056] S1-1, determining at least one video element in a target video according to text semantic features, wherein the at least one video element includes a first subject object;

[0057] S1-2, determining a posture change of the first subject in the target video based on the video reference features;

[0058] S1-3: Generate a target video based on posture changes of at least one video element and the first main object in the target video.

[0059] Optionally, in this embodiment, text semantic features are used to determine at least one video element displayed in the target video. For example, the content description text is "a puppy dancing on the football field". The text semantic features obtained by feature extraction of the content description text show that the video to be generated needs to have 1 puppy as the main object (first main object) and a football field as the video background. Both the puppy and the football field can be understood as video elements.

[0060] Optionally, in this embodiment, the posture change may refer to, but is not limited to, a change in the posture, position or shape of an object (such as an object or a human body) in the (target video) space, such as a displacement change (the position of the object in space changes, which may be a translation, rotation or staggered motion along a straight or curved path), a posture change (when the object is at rest or in motion, the posture of part or the whole of the body changes, such as bending, stretching, twisting, etc.), a shape change (the appearance of the object changes, such as a size change, deformation or expansion caused by compression or stretching), etc.

[0061] It should be noted that posture changes are usually the key attribute that determines whether a video is coherent, or whether the video presentation is coherent is closely related to posture changes. In this embodiment, in order to further improve the coherence of the target video, the video reference features are used to determine the posture changes of the first subject object in the target video, so that the posture changes in the target video are more consistent with the characteristics of the video itself, and the quality of the generated target video is higher.

[0062] To further illustrate, optionally, for example, the position distribution and element form of at least one video element on each video frame in the target video are determined, and based on the posture changes of the first main object in the target video, the position distribution and element form of the first main object on each video frame in the target video are dynamically and targetedly adjusted, so that the final presentation effect of the target video is not only limited to a collection of multiple frame images, but is more in line with the posture changes of the video characteristics, that is, a target video with higher coherence is presented.

[0063] Through the embodiments provided by the present application, at least one video element in a target video is determined based on text semantic features, wherein the at least one video element includes a first main object; the posture changes of the first main object in the target video are determined based on video reference features; and the target video is generated based on the posture changes of the at least one video element and the first main object in the target video, thereby achieving the purpose of presenting a target video with higher coherence, thereby realizing the technical effect of improving the video quality of the target video.

[0064] As an optional solution, feature extraction is performed on the content reference video to obtain video reference features, including:

[0065] Extracting features of a second subject object in the content reference video to obtain object performance features, wherein the object performance features are used to characterize posture changes of the second subject object in the content reference video, and the video reference features include the object performance features;

[0066] Determining a posture change of the first subject in the target video according to the video reference feature, including:

[0067] According to the object performance characteristics, a posture change of the first subject object in the target video is determined, wherein the posture change of the first subject object in the target video corresponds to the posture change of the second subject object in the content reference video.

[0068] Optionally, in this embodiment, the posture changes of the first subject object in the target video correspond to the posture changes of the second subject object in the content reference video. For example, the posture changes of the person (second subject object) during the dance in the content reference video 302 shown in FIG4 correspond to the posture changes of the puppy (first subject object) during the dance in the target video 308.

[0069] It should be noted that the video quality of the target video is improved by corresponding the posture changes between the existing content reference video and the target video to be generated. It can be understood that the posture changes of the first subject object in the target video are restored based on the posture changes of the second subject object in the content reference video, or the posture changes of the first subject object in the target video can be the first subject object performing a series of target actions in the target video, and this series of target actions can be the same or similar to the series of actions performed by the second subject object in the content reference video.

[0070] Through the embodiments provided by the present application, features of the second main object in the content reference video are extracted to obtain object expression features, wherein the object expression features are used to characterize the posture changes of the second main object in the content reference video, and the video reference features include object expression features; the posture changes of the first main object in the target video are determined based on the object expression features, wherein the posture changes of the first main object in the target video correspond to the posture changes of the second main object in the content reference video, thereby achieving the purpose of making the posture changes between the existing content reference video and the desired target video correspond to each other, thereby achieving the technical effect of improving the video quality of the target video, so that a series of actions performed by the first main object in the target video are referenced by a series of actions performed by the second main object in the content reference video.

[0071] As an optional solution, feature extraction is performed on the second main object in the content reference video to obtain object performance features, including:

[0072] S2-1, performing feature extraction on at least two target video frames containing a second subject object in the content reference video to obtain at least two object static features, wherein the object static features are used to characterize the position and shape of the second subject object in the target video frames;

[0073] S2-2, based on the temporal relationship information between at least two target video frames, integrate and process at least two object static features to obtain object dynamic features, wherein the object dynamic features are used to characterize the posture changes of the second main object in the content reference video, and the object performance features include object dynamic features.

[0074] It should be noted that the static features of the object are used to characterize the position and form of the second main object in the target video frame. The position and form themselves are static attributes, which are usually sufficient as the basis for image generation. However, if they are used as the basis for video generation, they lack dynamic attributes. This is because the coherence of the video is usually determined by the dynamic attributes, and without dynamic attributes, it is naturally impossible to generate high-quality videos.

[0075] Furthermore, in this embodiment, the object dynamic features are used to characterize the posture changes of the second main object in the content reference video. That is, this embodiment does not directly use the object static features as the basis for video generation, but uses the object static features to obtain the object dynamic features, and generates high-quality video based on the dynamic attributes contained in the object dynamic features.

[0076] To further illustrate, optionally, as shown in FIG5 , feature extraction is performed on at least two target video frames containing the second main object 504 in the content reference video 502 to obtain at least two object static features 506, wherein the object static features 506 are used to characterize the position and form of the second main object 504 in the target video frame; and the at least two object static features 506 are orderly integrated using the timing relationship information 508 between the at least two target video frames to obtain object dynamic features 510, wherein the object dynamic features 510 are used to characterize the posture changes of the second main object 504 in the content reference video 502.

[0077] Through the embodiments provided by the present application, feature extraction is performed on at least two target video frames containing a second main object in a content reference video to obtain at least two object static features, wherein the object static features are used to characterize the position and form of the second main object in the target video frame; the at least two object static features are integrated and processed using the temporal relationship information between the at least two target video frames to obtain object dynamic features, wherein the object dynamic features are used to characterize the posture changes of the second main object in the content reference video, and the object performance features include object dynamic features, thereby achieving the purpose of obtaining object dynamic features using object static features and generating high-quality video based on the dynamic attributes contained in the object dynamic features, thereby realizing the technical effect of improving the video quality of the target video.

[0078] As an optional solution, feature extraction is performed on at least two target video frames containing the second main object in the content reference video to obtain at least two object static features, including at least one of the following:

[0079] S3-1, extracting key points of a second subject object in at least two target video frames to obtain at least two key point features, wherein the key point features are used to characterize positions of key points of the second subject object in the target video frames, and the object static features include the key point features;

[0080] S3-2, performing keyline extraction on the second subject in the at least two target video frames to obtain at least two keyline features, wherein the keyline features are used to characterize the positions of the keylines of the second subject in the target video frames, and the object static features include the keyline features;

[0081] S3-3, performing contour extraction on the second subject in the at least two target video frames to obtain at least two contour features, wherein the contour features are used to characterize the morphological position of the contour of the second subject in the target video frames, and the object static features include the contour features;

[0082] S3-4, performing edge extraction on the second main object in the at least two target video frames to obtain at least two first object features, wherein the object static features include the first object features;

[0083] S3-5, performing depth extraction on the second main object in the at least two target video frames to obtain at least two second object features, wherein the object static features include the second object features;

[0084] S3-6, performing white mold extraction on the second main object in at least two target video frames to obtain at least two third object features, wherein the object static features include the third object features.

[0085] Optionally, in this embodiment, key point extraction may refer to, but is not limited to, automatically detecting and locating important feature points from an image. These feature points typically have significant structural, texture, or shape information. For example, feature detection algorithms such as Harris corner detection, SIFT (Scale Invariant Feature Transform), and SURF (Super Rapid Robust Feature Recognition) may be used to find key points in an image. These algorithms may determine key points based on features such as the local structure, gradient direction, and scale variation of the image.

[0086] Optionally, in this embodiment, keyline extraction may refer to, but is not limited to, extracting lines with important visual information from an image. These lines are typically primary contours, boundaries, or other important linear structures in the image. For example, lines with visual significance and importance may be screened based on criteria such as line length, curvature, and line fit. Curvature calculations, line fitting algorithms, and the like may be used to assess line quality and importance.

[0087] Optionally, in this embodiment, contour extraction can be, but is not limited to, extracting the boundary contour of an object from an image. A contour can be considered as a line segment connecting discontinuous points on the surface of an object, and can represent the shape and structural information of the object. For example, based on a binary image, a contour extraction algorithm can be used to detect and extract the boundary contour of the object, specifically using a contour detection algorithm based on edge connection (such as the Moore-Neighbor algorithm, the kNN detection algorithm, etc.) and a region growing algorithm based on pixel connection.

[0088] Optionally, in this embodiment, edge extraction can be used, but is not limited to, to detect and extract edge information of objects in an image. An edge generally represents a sudden change or discontinuity in brightness, color, texture, etc. in an image, and is a boundary between objects or between an object and a background. Implementation methods include Canny edge detection (obtaining high-quality edge results through a multi-step process, including Gaussian filtering, calculating image gradients, non-maximum suppression, double thresholding, and edge connection), Sobel operator (performing convolution operations on the horizontal and vertical directions of the image, respectively, to obtain two gradient images, and by combining these two gradient images, an edge intensity image can be obtained), Laplacian operator (by performing Laplacian filtering operations on the image, an edge image can be obtained), adaptive thresholding (segmenting the image into small regions, and selecting a threshold in each region based on local statistical characteristics), etc.

[0089] Optionally, in this embodiment, depth extraction may refer to, but is not limited to, a process of obtaining depth information from an image or scene. Depth information represents the distance between different points in an object or scene and a camera. Implementation methods include facial depth extraction (using devices such as infrared cameras, structured light, or time-of-flight sensors to obtain depth information of the facial area), binocular stereo vision (estimating the depth of a scene through images from two perspectives, based on the parallax between the left and right eye images, that is, the offset between corresponding pixels, thereby inferring the distance of the object), three-dimensional reconstruction (restore the geometric structure of a three-dimensional scene through multiple images or video sequences, and use multi-view geometry, structured light projection, optical flow estimation and other technologies for depth extraction and stereo reconstruction), deep learning methods (by using structures such as deep convolutional neural networks, depth information can be learned and predicted directly from a single image), etc.

[0090] Optionally, in this embodiment, the white mold may refer to, but is not limited to, a prototype model of a building, product, or sculpture, etc., which is used as a reference for making a formal product or structure, and may be a model made of materials such as wood, clay, or polymer.

[0091] It should be noted that a variety of extraction methods for subject objects are provided, and corresponding extraction methods can be flexibly used according to actual application scenarios and processing objects to meet the needs of corresponding scenarios and processing objects, thereby accurately determining the static object features of the second subject object in the target video frame.

[0092] As an optional solution, a target video is generated based on text semantic features and video reference features, including:

[0093] The text semantic features and video reference features are input into a video generation model to obtain a target video output by the video generation model, wherein the video generation model is a neural network model trained based on multiple video sample data and used to generate videos.

[0094] It should be noted that in order to improve the generation efficiency of the target video, a video generation model is used to generate the corresponding target video, wherein the video generation model can be but is not limited to a video diffusion model, which generates video data by denoising from random noise step by step.

[0095] To further illustrate, as shown in FIG6 , encoders A and B are optionally used to extract features from the acquired content description text 602-1 and content reference video 602-2, generating text semantic features 604-1 and video reference features 604-2. Furthermore, these text semantic features 604-1 and video reference features 604-2 are input into a video generation model 606 , processed by the video generation model 606 to generate video features. The video features are then converted into video data for a target video 608 through a decoder.

[0096] Through the embodiments provided in the present application, text semantic features and video reference features are input into a video generation model to obtain a target video output by the video generation model, wherein the video generation model is a neural network model for generating videos obtained by training based on multiple video sample data, thereby achieving the purpose of generating the desired target video using the video generation model, thereby realizing the technical effect of improving the generation efficiency of the target video.

[0097] As an optional solution, before inputting the text semantic features and the video reference features into the video generation model, the method further includes:

[0098] S4-1, obtaining an image generation model, wherein the image generation model is a neural network model trained based on a plurality of image sample data and used to generate images;

[0099] S4-2, adjusting the image generation model to obtain an initial video generation model, where the initial video generation model consists of a convolutional layer and an attention layer that can process temporal dimension information;

[0100] S4-3, training the initial video generation model based on multiple video sample data to obtain a video generation model.

[0101] It should be noted that if the video generation model used to generate the desired video is directly trained, it usually requires massive training data and high computing power to provide support. Regardless of cost or efficiency, the training quality of the video generation model cannot be guaranteed within a reasonable range.

[0102] In this embodiment, a more mature image generation model is adjusted to make the adjusted image generation model (video generation model) suitable for video generation. Prior to the modification, the image generation model is trained with an appropriate number of image training samples. Once the trained image generation model is obtained, the adjusted image generation model is then trained with an appropriate number of video training samples.

[0103] To further illustrate, optionally, assuming that the convolution layer of the image generation model is 3×3, the original convolution layer is expanded to a 1×3×3 convolution layer through adjustment, and a temporal attention layer is added to the video generation model to enhance the model's understanding of sequence frames and the stability of generating continuous frames of the video, so as to adapt to the video generation process.

[0104] Through the embodiments provided in the present application, an image generation model is obtained, wherein the image generation model is a neural network model for generating images obtained by training based on multiple image sample data; the image generation model is adjusted to obtain an initial video generation model, wherein the initial video generation model is composed of a convolutional layer and an attention layer that can process temporal dimension information; the initial video generation model is trained using multiple video sample data to obtain a video generation model, thereby achieving the purpose of adjusting the video generation model based on a more mature image generation model, thereby realizing the technical effect of ensuring the training quality of the video generation model within a reasonable range.

[0105] As an optional solution, text semantic features and video reference features are input into the video generation model to obtain the target video output by the video generation model, including:

[0106] Calling a single graphics processor unit to run a video generation model to process the input text semantic features and video reference features to obtain the target video output by the video generation model;

[0107] After calling a single graphics processor unit to run a video generation model to process input text semantic features and video reference features to obtain a target video output by the video generation model, the method further includes:

[0108] Inserting the associated video frame into the video frame sequence corresponding to the target video to obtain a new video, wherein the video length corresponding to the new video is greater than the video length corresponding to the target video.

[0109] Optionally, in this embodiment, a graphics processing unit (GPU) is a type of hardware specifically designed for processing graphics and parallel computing tasks. GPUs have a large number of cores and high-speed memory, enabling them to handle multiple data operations simultaneously and execute large-scale computing tasks concurrently. GPUs excel at processing intensive computing tasks such as large data sets, matrix operations, image processing, and simulations.

[0110] Optionally, in this embodiment, inserting associated video frames can be understood as, but not limited to, adding additional frames to the target video. By adding additional frames to the target video, the frame rate of the target video is changed. This can increase the length of the output new video and make the motion of the new video smoother. Specifically, adding additional frames to the target video can be achieved through, but not limited to, linear interpolation, optical flow estimation, frame blending, etc.

[0111] Linear interpolation distributes pixel values ​​evenly over time by interpolating between adjacent frames. Optical flow estimation estimates the pixel values ​​of the intermediate frame by analyzing the optical flow between two frames based on the motion characteristics of pixels. Blended frames consider the movement and deformation of objects in a video sequence and generate interpolated frames by sampling and blending multiple adjacent frames.

[0112] It should be noted that to improve the coherence of the target video, processing can be performed using, but is not limited to, a single graphics processing unit. However, the performance of a single graphics processing unit is limited, and the amount of video data output is naturally also limited. For example, only shorter target videos can be output. Shorter target videos generally do not meet user needs. In this embodiment, interpolation is employed to address this limitation.

[0113] Through the embodiments provided in the present application, a single graphics processor unit is called, and a video generation model is run to process the input text semantic features and video reference features to obtain a target video output by the video generation model; associated video frames are inserted into the video frame sequence corresponding to the target video to obtain a new video, wherein the video length corresponding to the new video is greater than the video length corresponding to the target video, that is, the interpolation method is used to compensate for the defect of the shorter length of the generated video caused by calling a single graphics processor unit, thereby achieving the technical effect of improving the coherence of the target video.

[0114] As an alternative solution, for ease of understanding, the above video generation method is applied to the scenario of generating character videos based on text. In existing technical solutions, videos generated using two-dimensional (2D) painting techniques often have significant inter-frame jitter and require a lot of computing time. Three-dimensional (3D) video generation methods face the dependence on massive amounts of data and computing power, are extremely costly, difficult to deploy, and have poor stability in the generated results.

[0115] This embodiment, by extending the 2D diffusion model to the 3D video generation network and combining it with the Controlnet control model, can greatly reduce the data and computing power costs required to train the 3D video generation model, achieving the goal of being driven by a single data card. At the same time, the reuse of the 2D diffusion model ensures the quality of the generated video, and the expansion of the temporal attention module can effectively maintain the stability of the video. Compared with previous solutions, this embodiment can make up for the shortcomings in both effect and efficiency, and can generate high-quality video content with ultra-low training and inference costs, and can be widely used in video generation tasks of various categories. In addition, this embodiment can also be applied to video content of other categories besides characters. This is only an example and is not limited.

[0116] Optionally, this embodiment constructs a video generation model based on the expansion of the 2D logical data model (Latent Diffusion Model, LDM for short), which can greatly reduce the data and computing power costs required for model training, and at the same time can effectively reuse the generation capabilities of the existing massive 2D painting models for video creation. At the same time, this embodiment also integrates control capabilities into the video generation model, extracts gestures from the reference video to control the content of the generated video, thereby effectively enhancing the stability and controllability of the generated video content. Furthermore, this embodiment combines the above-mentioned video generation technology with an interpolation algorithm to generate an initial video with a low frame rate by extracting key frames, and then improves the video frame rate through an interpolation algorithm based on a neural network, thereby greatly improving the efficiency and smoothness of video generation.

[0117] Optionally, in this embodiment, the video generation model and the posture control signal are fused to support users to input content reference videos, and the video content is generated in combination with the content description text, which can effectively improve the controllability and generation quality of the generated video content. By extracting skeletal animation sequences from differentiated content reference videos and combining them with diverse content description texts, this embodiment can achieve refined control of video content and output video content of different styles, roles, and scenes. Furthermore, this embodiment can generate high-quality videos within minutes through a single GPU. This efficient, low-cost video generation solution has great application prospects in the fields of video and film production, game PV, and animation production, and can meet the diverse and customized video production needs.

[0118] To further illustrate, as shown in Figure 7, the initial input of this embodiment is optionally a user-provided content reference video and a text description of the content. The content reference video is used to extract gestures and movements, thereby controlling the character movements in the generated video, while the content description text is used to control the specific content of the generated video, such as the character image, style, and background. The extracted gestures and text semantic information are uniformly fed into the video diffusion model to generate the initial video. This initial video can then be subjected to post-processing methods such as video interpolation and super-resolution reconstruction to generate the final video effect.

[0119] It should be noted that the present embodiment can be used to generate character-related video content, and the input content reference video should contain character movements, so as to extract gestures to control the content of the generated video. Specifically, the present embodiment can, but is not limited to, use the openpose algorithm to extract human key points (18 key points including the head and body) from each frame of the content reference video, and visualize the key points into skeletal connection relationships to obtain a converted gesture action graph. The converted gesture action graph will be used as the input of Controlnet to guide the generation of video content. Due to the limited video memory of a single GPU, there is an upper limit to the number of video frames generated at a time (a single A100 GPU can generate 70 frames of video each time). In order to improve the duration of the generated video, the present embodiment can extract an 8fps key frame sequence from the content reference video for action sequence extraction, aiming to improve processing efficiency and the duration of the generated video.

[0120] To further illustrate, optionally, as shown in FIG8 , an image 802 of the current video frame is determined from a character video (i.e., a content reference video including the character), and then key points of the human body are extracted from the image 802 , and the key points are visualized as skeletal connection relationships to obtain a converted posture action diagram 804 .

[0121] Optionally, as shown in Figure 9, the video diffusion model of this embodiment is composed of four parts, including a latent variable decoder, a text encoder, a video diffusion model, and a posture control module. First, the text encoder is used to extract text semantic features from the input content description text, and the posture control module helps to extract video reference features, i.e., action features, from the content reference video. Specifically, this embodiment follows the stable diffusion model, and uses CLIP Text Encoder for text encoding, while using Controlnet as a posture control module. Then, the text semantic features and video reference features are simultaneously input into the video diffusion model for calculation, thereby obtaining the latent variable features of the generated video. The latent variable features will eventually be converted into video content through processing by the latent variable decoder.

[0122] The video diffusion model inherits the basic network structure of the traditional 2D diffusion model (LDM). This structure uses U-Net as its foundation and utilizes a network consisting of 2D convolutional layers, upsampling layers, downsampling layers, and a self-attention mechanism to process input features. This traditional 2D network structure is limited to generating single images. To directly generate video content, this embodiment extends the 2D diffusion model to a 3D video generation model. Specifically, this embodiment expands the original 3×3 convolutional layers in the U-Net to 1×3×3 convolutional layers and adds a temporal attention layer to enhance the model's understanding of sequential frames and the stability of generating continuous video frames. Before deployment, this embodiment can train the network using a small amount of video data. During training, the parameters of the original 2D U-Net remain fixed, and only the newly added temporal attention module is trained. This design not only fully utilizes existing 2D model resources to generate videos of different styles and concepts, but also, by fixing the original 2D diffusion model parameters during model training, only a minimal amount of data is required to enable the 3D video generation model to generate videos.

[0123] In order to further strengthen the control ability for video content, the present embodiment further embeds the posture control module into the video generation system to output highly controllable video content. Specifically, the present embodiment adopts the Controlnet model, which can realize motion control of the drawn image content through the input of skeletal movements in 2D AI painting. In this system, the present embodiment inputs the 8fps skeleton sequence frames extracted from the content reference video into the posture control module, which processes the skeleton sequence frame by frame and splices the multi-frame features together to guide the calculation of the video generation model. The output of the model is the latent variable feature of the video content, which will eventually be converted into an initial video with a low frame rate (8fps) by a latent variable decoder.

[0124] Optionally, to further improve video smoothness, this embodiment uses video interpolation to increase the frame rate of the generated initial video to 32fps. Specifically, this embodiment first breaks the generated initial video into individual frames. A neural network-based FILM algorithm is then used to perform a 4x interpolation process on the frame sequence. The newly generated frames are then combined into a high-frame-rate video, significantly improving smoothness.

[0125] Through the embodiments provided in this application, the 2D diffusion model is expanded to a 3D video generation model, and combined with the Controlnet control model, the data and computing power costs required to train the 3D video generation model can be greatly reduced, and a single data card can be driven. At the same time, this reuse based on the 2D diffusion model ensures the quality of the generated video, and the expansion of the temporal attention module can effectively maintain the stability of the video. Compared with previous solutions, this embodiment makes up for the shortcomings in both effect and efficiency. It can generate high-quality video content with ultra-low training and inference costs, and can be widely used in video generation tasks of various categories.

[0126] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0127] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0128] According to another aspect of the embodiments of the present application, a video generation device for implementing the above-mentioned video generation method is also provided. As shown in FIG10 , the device includes:

[0129] A first acquisition unit 1002 is configured to acquire a content description text and a content reference video, wherein the content description text includes information describing the target content expressed by the target video to be generated, and the content reference video includes action reference information related to the target content;

[0130] Extraction unit 1004 is configured to perform feature extraction on the content description text to obtain text semantic features, wherein the text semantic features are used to represent semantic information of the content description text; perform feature extraction on the content reference video to obtain video reference features, wherein the video reference features are used to represent action reference information in the content reference video;

[0131] The generating unit 1006 is configured to generate a target video according to text semantic features and video reference features.

[0132] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation device, which will not be described in detail in this example.

[0133] As an optional solution, the generating unit 1006 includes:

[0134] A first determining module is configured to determine at least one video element in a target video according to text semantic features, wherein the at least one video element includes a first subject object;

[0135] A second determining module is used to determine the posture change of the first subject in the target video based on the video reference features;

[0136] The acquisition module is used to acquire a target video based on posture changes of at least one video element and a first main object in the target video.

[0137] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0138] As an optional solution, the extraction unit 1004 includes:

[0139] an extraction module configured to extract features of a second subject object in the content reference video to obtain object performance features, wherein the object performance features are used to characterize posture changes of the second subject object in the content reference video, and the video reference features include the object performance features;

[0140] The generating unit 1006 includes:

[0141] The third determining module is used to determine the posture change of the first main object in the target video according to the object performance characteristics, wherein the posture change of the first main object in the target video corresponds to the posture change of the second main object in the content reference video.

[0142] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0143] As an optional solution, extract the module, including:

[0144] an extraction submodule, configured to perform feature extraction on at least two target video frames containing the second subject object in the content reference video to obtain at least two object static features, wherein the object static features are used to characterize the position and morphology of the second subject object in the target video frames;

[0145] The processing submodule is used to integrate the static features of at least two objects based on the temporal relationship information between at least two target video frames to obtain the dynamic features of the object, wherein the dynamic features of the object are used to characterize the posture changes of the second main object in the content reference video, and the object performance features include the dynamic features of the object.

[0146] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0147] As an optional solution, extracting submodules includes at least one of the following:

[0148] a first extraction subunit, configured to extract key points of a second subject object in at least two target video frames to obtain at least two key point features, wherein the key point features are used to characterize positions of key points of the second subject object in the target video frames, and the object static features include the key point features;

[0149] a second extraction subunit, configured to extract key lines of the second subject object in the at least two target video frames to obtain at least two key line features, wherein the key line features are used to characterize the positions of the key lines of the second subject object in the target video frames, and the object static features include the key line features;

[0150] a third extraction subunit, configured to extract the contour of the second subject object in the at least two target video frames to obtain at least two contour features, wherein the contour features are used to characterize the morphological position of the contour of the second subject object in the target video frames, and the object static features include the contour features;

[0151] a fourth extraction subunit, configured to perform edge extraction on the second main object in the at least two target video frames to obtain at least two first object features, wherein the object static features include the first object features;

[0152] a fifth extraction subunit, configured to perform depth extraction on a second main object in at least two target video frames to obtain at least two second object features, wherein the object static features include the second object features;

[0153] The sixth extraction subunit is configured to perform white mold extraction on the second main object in at least two target video frames to obtain at least two third object features, wherein the object static features include the third object features.

[0154] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0155] As an optional solution, the generating unit 1006 includes:

[0156] The input module is used to input text semantic features and video reference features into the video generation model to obtain the target video output by the video generation model, wherein the video generation model is a neural network model trained using multiple video sample data and used to generate videos.

[0157] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0158] As an optional solution, the device further includes:

[0159] a second acquisition unit, configured to acquire an image generation model before inputting the text semantic features and the video reference features into the video generation model, wherein the image generation model is a neural network model trained using a plurality of image sample data and used to generate images;

[0160] A determination unit is used to adjust the image generation model to obtain an initial video generation model, wherein the initial video generation model is composed of a convolution layer and an attention layer that can process temporal dimension information;

[0161] The training unit is used to train the initial video generation model based on multiple video sample data to obtain the video generation model.

[0162] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0163] As an optional solution, the input module includes:

[0164] An input submodule is used to call a single graphics processor unit to run a video generation model to process the input text semantic features and video reference features to obtain a target video output by the video generation model;

[0165] The device also includes: a frame insertion submodule, which is used to call a single graphics processor unit, run a video generation model to process text semantic features and video reference features, obtain a target video output by the video generation model, and then insert associated video frames into a video frame sequence corresponding to the target video to obtain a new video, wherein the video length corresponding to the new video is greater than the video length corresponding to the target video.

[0166] For specific embodiments, reference may be made to the examples shown in the above-mentioned video generation method, which will not be described in detail in this example.

[0167] According to another aspect of the embodiments of the present application, an electronic device for implementing the above-mentioned video generation method is also provided. The electronic device may be, but is not limited to, the user device 102 or the server 112 shown in Figure 1. This embodiment takes the electronic device as the user device 102 as an example. Further, as shown in Figure 11, the electronic device includes a memory 1102 and a processor 1104. A computer program is stored in the memory 1102, and the processor 1104 is configured to execute the steps in any of the above-mentioned method embodiments through the computer program.

[0168] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0169] Alternatively, those skilled in the art will appreciate that the structure shown in FIG11 is merely illustrative and does not limit the structure of the electronic device. For example, the electronic device may include more or fewer components (such as network interfaces) than those shown in FIG11, or may have a configuration different from that shown in FIG11.

[0170] Memory 1102 can be used to store software programs and modules, such as program instructions / modules corresponding to the video generation method and apparatus in the embodiments of the present application. Processor 1104 executes the software programs and modules stored in memory 1102 to perform various functional applications and data processing, thereby implementing the aforementioned video generation method. Memory 1102 can include high-speed random access memory (RAM) and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, memory 1102 can further include memory remotely located from processor 1104, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. Memory 1102 can specifically, but is not limited to, be used to store information such as content description text, content reference videos, and target videos. As an example, as shown in FIG11 , memory 1102 can include, but is not limited to, the first acquisition unit 1002, extraction unit 1004, and generation unit 1006 of the aforementioned video generation apparatus. In addition, it may also include but is not limited to other module units in the above-mentioned video generation device, which will not be repeated in this example.

[0171] Optionally, the transmission device 1106 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 1106 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 1106 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0172] In addition, the electronic device further includes: a display 1108 for displaying information such as the content description text, content reference video, and target video; and a connection bus 1110 for connecting various module components in the electronic device.

[0173] In other embodiments, the user device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes via network communication. The nodes may form a peer-to-peer network, and any computing device, such as a server, user device, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0174] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instructions containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component and / or installed from a removable medium. When the computer program is executed by a central processing unit, the various functions provided in the embodiments of the present application are performed.

[0175] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0176] It should be noted that the computer system of the electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0177] A computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) or programs loaded from the storage unit into random access memory (RAM). The RAM also stores various programs and data required for system operation. The CPU, the read-only memory, and the RAM are connected to each other via a bus. Input / output interfaces (I / O interfaces) are also connected to the bus.

[0178] The following components are connected to the input / output interface: an input section including a keyboard, mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section including a hard disk; and a communication section including a network interface card such as a local area network card and a modem. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. Removable media such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc. are installed in the drive as needed so that computer programs read from them can be installed into the storage section as needed.

[0179] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication portion, and / or installed from a removable medium. When the computer program is executed by a central processing unit, the various functions defined in the system of the present application are performed.

[0180] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations described above.

[0181] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the electronic device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0182] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0183] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0184] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed user equipment can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0186] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0188] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video generation method, performed by an electronic device, comprising: Acquire a content description text and a content reference video, wherein the content description text includes information describing a target content expressed by a target video to be generated, and the content reference video includes action reference information related to the target content; Performing feature extraction on the content description text to obtain text semantic features, wherein the text semantic features are used to represent semantic information of the content description text; performing feature extraction on the content reference video to obtain video reference features, wherein the video reference features are used to represent the action reference information in the content reference video; The target video is generated according to the text semantic features and the video reference features.

2. The method according to claim 1, wherein generating the target video according to the text semantic features and the video reference features comprises: Determine at least one video element in the target video according to the text semantic feature, wherein the at least one video element includes a first subject object; Determining, according to the video reference features, a change in the posture of the first subject object in the target video; The target video is generated based on the posture changes of the at least one video element and the first main object in the target video.

3. The method according to claim 2, wherein extracting features from the content reference video to obtain video reference features comprises: Extracting features of the second subject object in the content reference video to obtain object performance features, wherein the object performance features are used to characterize posture changes of the second subject object in the content reference video, and the video reference features include the object performance features; The step of determining, based on the video reference feature, a posture change of the first subject object in the target video includes: According to the object performance characteristics, a posture change of the first main object in the target video is determined, wherein the posture change of the first main object in the target video corresponds to the posture change of the second main object in the content reference video.

4. The method according to claim 3, wherein extracting features of the second main object in the content reference video to obtain object performance features comprises: Performing feature extraction on at least two target video frames containing the second main object in the content reference video to obtain at least two object static features, wherein the object static features are used to characterize the position and form of the second main object in the target video frames; Based on the temporal relationship information between the at least two target video frames, the static features of the at least two objects are integrated to obtain the dynamic features of the object, wherein the dynamic features of the object are used to characterize the posture changes of the second main object in the content reference video, and the object expression features include the dynamic features of the object.

5. The method according to claim 4, wherein the feature extraction of at least two target video frames containing the second main object in the content reference video to obtain at least two object static features comprises at least one of the following: Extract key points of the second subject object in the at least two target video frames to obtain at least two key points. Key point features, where The key point feature is used to characterize the position of the key point of the second subject object in the target video frame, and the object static feature includes the key point feature; Extracting key lines of the second subject object in the at least two target video frames to obtain at least two key line features, wherein the key line features are used to characterize the positions of key lines of the second subject object in the target video frames, and the object static features include the key line features; Extracting the contour of the second main object in the at least two target video frames to obtain at least two contour features, wherein the contour features are used to characterize the morphological position of the contour of the second main object in the target video frames, and the object static features include the contour features; Performing edge extraction on the second main object in the at least two target video frames to obtain at least two first object features, wherein the object static features include the first object features; Performing depth extraction on the second main object in the at least two target video frames to obtain at least two second object features, wherein the object static features include the second object features; Perform white mold extraction on the second main object in the at least two target video frames to obtain at least two third object features, wherein the object static features include the third object features.

6. The method according to any one of claims 1 to 5, wherein generating the target video according to the text semantic features and the video reference features comprises: The text semantic features and the video reference features are input into a video generation model to obtain the target video output by the video generation model, wherein the video generation model is a neural network model trained based on multiple video sample data and used to generate videos.

7. The method according to claim 6, before inputting the text semantic features and the video reference features into the video generation model, the method further comprises: Acquire an image generation model, wherein the image generation model is a neural network model trained based on a plurality of image sample data and used to generate images; For the image generation model, an initial video generation model is obtained, wherein the initial video generation model is composed of a convolutional layer and an attention layer that can process temporal dimension information; The initial video generation model is trained based on the multiple video sample data to obtain the video generation model.

8. The method according to claim 6 or 7, wherein the inputting the text semantic features and the video reference features into a video generation model to obtain the target video output by the video generation model comprises: Calling a single graphics processor unit to run the video generation model to process the text semantic features and the video reference features to obtain the target video output by the video generation model; After calling a single graphics processor unit and running the video generation model to process the text semantic features and the video reference features to obtain the target video output by the video generation model, the method further includes: Inserting associated video frames into a video frame sequence corresponding to the target video to obtain a new video, wherein a video length corresponding to the new video is greater than a video length corresponding to the target video.

9. A video generating device, comprising: A first acquisition unit is used to acquire a content description text and a content reference video, wherein the content description text includes information describing a target content expressed by a target video to be generated, and the content reference video includes action reference information related to the target content; An extraction unit is used to perform feature extraction on the content description text to obtain text semantic features, wherein the text semantic features are used to represent the semantic information of the content description text; perform feature extraction on the content reference video to obtain video reference features, wherein the video reference features are used to represent the action reference information in the content reference video; A generating unit is used to generate the target video according to the text semantic features and the video reference features.

10. A computer-readable storage medium, the computer-readable storage medium comprising a stored program, wherein: When the program is executed by an electronic device, the method described in any one of claims 1 to 8 is executed.

11. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images