Video generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video generation method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. According to the method, multiple image features such as semantic features, implicit features and masking features extracted from an original image and camera features serve as input of a first video generation model, so that the first video generation model can display a target object in the original image in a dynamic video; and the first video generation model can learn the camera features, so that the camera track and the camera view angle of the dynamic video better conform to the display logic. Compared with a traditional mode of directly inputting the semantic features into the first video generation model, the method has the advantages that various image features and camera features are introduced, so that the first video generation model takes the target object as a main body, the dynamic video which is richer in details and better conforms to the display logic is generated, and the video quality and the availability rate are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generation method, device, electronic device and storage medium. Background Art

[0002] In the case of object display, compared with directly displaying the object through static images, displaying the object through dynamic video can better capture the user's attention and attract the user's sight, and dynamic video can better convey the object information to the user. However, when faced with a large number of objects, batch production of dynamic videos of the object through shooting is not only difficult, but also inefficient. Therefore, how to quickly obtain dynamic videos of object objects is a problem that needs to be solved.

[0003] At present, the commonly used method is to use the semantic features of the original image to guide the processing process of the neural network model when the original image of the object is available, so that the neural network model generates a dynamic video associated with the semantic features.

[0004] However, since current neural network models tend to simulate the natural movement of objects in the real world, it is usually difficult for the neural network model to generate dynamic videos that conform to the display logic of the object. In other words, the video generation effect of the above method is not good, and the usability of the dynamic videos generated by the neural network model is low. Summary of the invention

[0005] The present disclosure provides a video generation method, device, electronic device and storage medium. By introducing multiple image features and camera features, the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more in line with the display logic, and improves the video quality and availability. The technical solution of the present disclosure is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a video generation method is provided, the method comprising:

[0007] Acquire a camera feature associated with an original image, wherein the original image includes a target object, and the camera feature is used to represent a trajectory and a viewing angle of the camera;

[0008] Inputting the original image into a first feature extraction model for feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from the portion of the original image other than the target object;

[0009] The semantic features, the latent features, the masked features and the camera features are input into a first video generation model for decoding and encoding to obtain a dynamic video, where the dynamic video is used to display the target object based on the camera features.

[0010] According to another aspect of an embodiment of the present disclosure, a video generating device is provided, the device comprising:

[0011] a determining unit configured to obtain a camera feature associated with an original image, wherein the original image includes a target object, and wherein the camera feature is used to represent a trajectory and a viewing angle of a camera;

[0012] an extraction unit configured to input the original image into a first feature extraction model to perform feature extraction, and obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from a portion of the original image other than the target object;

[0013] The generation unit is configured to input the semantic features, the latent features, the masked features and the camera features into a first video generation model for decoding and encoding to obtain a dynamic video, wherein the dynamic video is used to display the target object based on the camera features.

[0014] In some embodiments, the determination unit is configured to determine the camera feature associated with the item type as the camera feature associated with the original image based on the item type corresponding to the original image, where the item type is the type of the target object contained in the original image; or, in a case where multiple preset camera features have been set, in response to a selection operation on any preset camera feature, determine the preset camera feature as the camera feature associated with the original image, and different preset camera features are used to represent different camera trajectories and viewing angles.

[0015] In some embodiments, the generating unit comprises:

[0016] A fusion subunit is configured to perform feature fusion on the latent feature, the masked feature and the noise data to obtain an intermediate feature;

[0017] A generation subunit is configured to use the intermediate features as input data of the input layer of the first video generation model, use the camera features and the semantic features as data for feature fusion in each layer of the first video generation model, and perform encoding and decoding based on the first video generation model to obtain a dynamic video of the target object.

[0018] In some embodiments, the fusion subunit is configured to copy the latent features, the masked features and the noise data respectively to obtain a target number of latent features, the target number of masked features and the target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated; for the target number of latent features, the target number of masked features and the target number of noise data, the target number of intermediate features are obtained for the latent features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame.

[0019] In some embodiments, the apparatus further comprises:

[0020] an acquisition unit, configured to acquire a plurality of training data sets, the plurality of training data sets comprising a first data set, a second data set and a third data set, the first data set comprising a plurality of sample videos, the second data set comprising sample videos of sample objects in a target domain, the target domain being a domain to which the target objects belong, and the third data set comprising sample videos in the second data set that meet a preset condition;

[0021] The training unit is configured to train the second feature extraction model and the second video generation model based on the first data set, the second data set and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

[0022] In some embodiments, the training unit is configured to perform a first-stage training on the second feature extraction model and the second video generation model based on the first data set and the second data set to obtain a first-stage feature extraction model and a first-stage video generation model, wherein the dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set; perform a second-stage training on the first-stage feature extraction model and the first-stage video generation model based on the second data set to obtain a second-stage feature extraction model and a second-stage video generation model, wherein the second-stage video generation model is used to generate dynamic video corresponding to the target field; perform a third-stage training on the second-stage feature extraction model and the second-stage video generation model based on the third data set to obtain the first feature extraction model and the first video generation model, wherein the video score of the dynamic video output by the third-stage video generation model is higher than the video score of the dynamic video output by the second-stage video generation model.

[0023] In some embodiments, the training unit is configured to, for any training stage, input the sample video in the training data set into the current video processing model for feature extraction to obtain sample camera features, wherein the sample camera features are used to represent the trajectory and viewing angle of the camera of the sample video; input the first frame of the sample video image into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features and sample masking features, wherein the sample latent features are used to represent the potential information of the first frame of the video image, and the sample masking features are used to distinguish the sample object in the first frame of the video image and the part of the first frame of the video image other than the sample object; input the sample semantic features, the sample latent features, the sample masking features and the sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, wherein the predicted video is used to display the sample object based on the sample camera features; based on the error between the predicted video and the sample video, update the parameters of the current feature extraction model, the current video generation model and the current video processing model through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

[0024] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0025] one or more processors;

[0026] A memory for storing program codes executable by the processor;

[0027] The processor is configured to execute the program code to implement the above-mentioned video generation method.

[0028] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned video generating method.

[0029] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned video generating method is implemented.

[0030] The disclosed embodiment provides a video generation method, in which multiple image features and camera features such as semantic features, implicit features, and masked features extracted from the original image are used as inputs to a first video generation model, so that the first video generation model can not only display the target object in the original image in the dynamic video, but also learn the camera features, so that the camera trajectory and camera perspective of the dynamic video are more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces multiple image features and camera features, so that the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, and improves the video quality and availability.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0033] Figure 1 It is a schematic diagram of an implementation environment of a video generation method according to an exemplary embodiment.

[0034] Figure 2 The figure is a flowchart of a video generating method according to an exemplary embodiment.

[0035] Figure 3 The figure is a flowchart of another video generating method according to an exemplary embodiment.

[0036] Figure 4 is a schematic diagram of an overall model according to an exemplary embodiment.

[0037] Figure 5 It is a block diagram of a video generating device according to an exemplary embodiment.

[0038] Figure 6 is a block diagram of another video generating device according to an exemplary embodiment.

[0039] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0042] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the original images involved in this disclosure are all obtained with full authorization.

[0043] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a video generation method according to an exemplary embodiment. Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0044] The terminal 101 may be a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or the like. An application may be installed and run on the terminal 101, and the application may generate a dynamic video based on an original image. The application is associated with the server 102, and the server 102 provides a background service to the terminal 101.

[0045] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0046] Server 102 is a server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, a distributed computing architecture is used between server 102 and terminal 101 for collaborative computing. Server 102 can be connected to terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0047] Figure 2 is a flowchart of a video generation method according to an exemplary embodiment. Figure 2 As shown, the method is executed by the terminal and includes the following steps:

[0048] In step S201, the terminal obtains camera features associated with an original image, where the original image includes a target object, and the camera features are used to represent a trajectory and a viewing angle of the camera.

[0049] In the disclosed embodiments, the original image includes the target object, and information such as the color, shape, and form of the target object can be obtained through the original image. For example, if the target object is a clothing item, the original image refers to a reference image of the clothing item. For ease of description, the camera feature can also be referred to as a camera movement feature. Different camera features correspond to different camera trajectories and camera perspectives.

[0050] In step S202, the terminal inputs the original image into the first feature extraction model for feature extraction to obtain semantic features, latent features and masked features. The latent features are used to represent the potential information of the original image, and the masked features are used to distinguish the target object in the original image and the part of the original image other than the target object.

[0051] In the disclosed embodiment, the first feature extraction model refers to a trained model for predicting dynamic video. Optionally, the model includes multiple modules, each of which is used to extract features of different dimensions or levels based on the original image. When the terminal extracts features from the original image through multiple modules in the model, optionally, multiple modules can be processed simultaneously, and multiple feature modules can also be processed in sequence according to a preset order, which is not limited here.

[0052] Among them, the implicit feature refers to the image feature determined based on the key information of the input image. The implicit feature describes the original image at a lower resolution than the original image. The masked feature is used to distinguish the target object area and the non-target object area in the original image through different pixel values.

[0053] In step S203, the terminal inputs the semantic features, implicit features, masked features and camera features into the first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

[0054] In the disclosed embodiment, the first video generation model generates multiple frames of video images based on various feature information and camera features in the original image, and then obtains a dynamic video based on the multiple frames of video images. The camera movement mode of the dynamic video refers to the camera movement mode indicated by the camera features learned by the model.

[0055] The disclosed embodiment provides a video generation method, in which multiple image features and camera features such as semantic features, implicit features, and masked features extracted from the original image are used as inputs to a first video generation model, so that the first video generation model can not only display the target object in the original image in the dynamic video, but also learn the camera features, so that the camera trajectory and camera perspective of the dynamic video are more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces multiple image features and camera features, so that the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, and improves the video quality and availability.

[0056] In some embodiments, obtaining camera features associated with the original image includes:

[0057] Based on the object type corresponding to the original image, the camera feature associated with the object type is determined as the camera feature associated with the original image, and the object type is the type of the target object contained in the original image; or,

[0058] In the case where multiple preset camera features are set, in response to a selection operation on any preset camera feature, the preset camera feature is determined as the camera feature associated with the original image, and different preset camera features are used to represent different camera trajectories and viewing angles.

[0059] In the disclosed embodiment, the terminal determines the camera features by the type of object corresponding to the original image, and can automatically match the camera features that are more suitable for the type of object, which is conducive to generating higher quality dynamic videos based on the camera features. The terminal provides a method for selecting preset camera features, which improves the flexibility of video generation and makes the video generation process more personalized. The above two methods enable the terminal to cope with a variety of application scenarios and needs, and ensure the effect and efficiency of video generation.

[0060] In some embodiments, inputting semantic features, latent features, masked features, and camera features into a first video generation model for decoding and encoding includes:

[0061] The latent features, masked features and noise data are fused to obtain intermediate features;

[0062] The intermediate features are used as input data of the input layer of the first video generation model, the camera features and semantic features are used as data for feature fusion in each layer of the first video generation model, encoding and decoding are performed based on the first video generation model to obtain a dynamic video of the target object.

[0063] In the disclosed embodiment, the terminal generates intermediate features based on implicit features and masked features combined with noise data, which can more accurately describe the contours and details of the target object, making the subsequently generated dynamic video more accurate and rich in details. At the same time, the terminal uses camera features and semantic features as data for feature fusion in the process of model-generated video, so that the model can adjust the video generation process and data according to the above features, making the video generation process more flexible. The above method improves the efficiency of video generation and is conducive to generating more accurate and higher-quality dynamic videos.

[0064] In some embodiments, the implicit features, masked features, and noise data are fused to obtain intermediate features, including:

[0065] The latent features, the masked features, and the noise data are respectively copied to obtain a target number of latent features, a target number of masked features, and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated;

[0066] For the target number of hidden features, the target number of masked features and the target number of noise data, the target number of intermediate features are obtained for the hidden features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame.

[0067] In the disclosed embodiment, the terminal copies implicit features, masked features and noise data to the target number so that each frame of video corresponds to an independent feature representation. This not only increases the controllability of the video generation process, but also makes the video generation for each frame based on more specific and rich feature information, which is conducive to generating more coherent dynamic video, thereby improving the quality and usability of the generated dynamic video.

[0068] In some embodiments, the training steps of the first feature extraction model and the first video generation model include:

[0069] Acquire multiple training data sets, the multiple training data sets include a first data set, a second data set and a third data set, the first data set includes multiple sample videos, the second data set includes sample videos of sample objects in a target domain, the target domain is the domain to which the target object belongs, and the third data set includes sample videos in the second data set that meet preset conditions;

[0070] Based on the first data set, the second data set and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

[0071] In the disclosed embodiment, the terminal performs model training through the first data set, so that the model can learn common camera features during the training process, thereby enhancing the generalization ability of the model. Then, the terminal fine-tunes the model through the second data set, so that the model can generate dynamic videos that are more in line with the characteristics of the target field, thereby improving the quality and relevance of the generated video. The terminal further fine-tunes the model through the third data set, so that the model can learn better camera features, which is conducive to further optimizing the effect and quality of the generated video. Through multi-stage training, the terminal enables the model to gradually and deeply learn the information of different data sets, thereby improving the overall training effect and quality. At the same time, since each stage is optimized for a specific training goal, the expected training effect can be achieved faster, thereby improving the efficiency of model training.

[0072] In some embodiments, based on the first data set, the second data set, and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, including:

[0073] Based on the first data set and the second data set, a second feature extraction model and a second video generation model are trained in the first stage to obtain a first-stage feature extraction model and a first-stage video generation model, wherein the dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set;

[0074] Based on the second data set, the feature extraction model of the first stage and the video generation model of the first stage are trained in the second stage to obtain the feature extraction model of the second stage and the video generation model of the second stage, and the video generation model of the second stage is used to generate dynamic videos corresponding to the target domain;

[0075] Based on the third data set, the feature extraction model of the second stage and the video generation model of the second stage are trained for the third stage to obtain the first feature extraction model and the first video generation model. The video score of the dynamic video output by the video generation model of the third stage is higher than the video score of the dynamic video output by the video generation model of the second stage.

[0076] In the disclosed embodiment, the terminal gradually trains in the first stage, the second stage, and the third stage, so that the model can gradually move from generalized learning to in-depth learning in a specific field, and then fine-tune and optimize the model. The above method is conducive to the model to better understand and adapt to the training objectives of different stages, which not only improves the overall training effect and quality, but also improves the efficiency of model training.

[0077] In some embodiments, based on the first data set, the second data set, and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, including:

[0078] For any training stage, the sample video in the training data set is input into the current video processing model for feature extraction to obtain sample camera features, which are used to represent the trajectory and viewing angle of the camera of the sample video;

[0079] Inputting the first frame of the sample video into the current feature extraction model for feature extraction, obtaining sample semantic features, sample latent features and sample masking features, wherein the sample latent features are used to represent potential information of the first frame of the video image, and the sample masking features are used to distinguish the sample object in the first frame of the video image and the part of the first frame of the video image other than the sample object;

[0080] Inputting sample semantic features, sample latent features, sample masking features and sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, which is used to display the sample object based on the sample camera features;

[0081] Based on the error between the predicted video and the sample video, the parameters of the current feature extraction model, the current video generation model and the current video processing model are updated through the objective function to obtain the feature extraction model of the training phase, the video generation model of the training phase and the video processing model of the training phase.

[0082] In the disclosed embodiment, at any training stage, the terminal simultaneously extracts the camera features of the sample video and the semantic features, implicit features, and masked features of the first frame of the video image, so that the model can obtain richer and more comprehensive information, which is conducive to maintaining the consistency of the video generation process. At the same time, the terminal introduces camera features during the training process so that the model can simulate the movement and behavior of the camera in the display scene of the target object, thereby improving the quality and usability of the generated video. In addition, the terminal evaluates the model performance by predicting the error between the video and the sample video, and continuously adjusts and optimizes the model parameters to improve the model performance.

[0083] Above Figure 2 The figure shows a flow chart of a video generation method of the present disclosure. The video generation scheme provided by the present disclosure is further described below. Figure 3 is a flowchart of another video generation method according to an exemplary embodiment. Figure 3 The method is executed by the terminal. The method comprises the following steps:

[0084] In step S301, the terminal obtains camera features associated with an original image, where the original image includes a target object, and the camera features are used to represent a trajectory and a viewing angle of the camera.

[0085] In the disclosed embodiment, the original image includes the target object, and the color, shape, form and other information of the target object can be obtained through the original image. For example, in a commodity display scenario, the target object is an object such as fruit, household items, clothing items, etc. Optionally, for clothing items, the target object may only include clothing items, or may also include a model wearing the clothing items, which is not limited here. Accordingly, the original image refers to a reference commodity image including the target object.

[0086] For ease of description, the camera features associated with the original image can also be called camera movement features. When a dynamic video of a target object is generated under the guidance of different camera features, the camera movement of the target object in the dynamic video is not exactly the same, which is manifested in different camera trajectories and camera perspectives.

[0087] In some embodiments, the terminal obtains the camera features associated with the original image in a variety of ways. Accordingly, based on the object type corresponding to the original image, the terminal determines the camera features associated with the object type as the camera features associated with the original image, where the object type is the type of the target object contained in the original image; or, in the case where multiple preset camera features have been set, the terminal determines the preset camera features as the camera features associated with the original image in response to a selection operation of any preset camera feature, where different preset camera features are used to represent different camera trajectories and viewing angles.

[0088] For example, if the object type corresponding to the original image is a large appliance, the camera feature associated with this object type instructs the camera to display the target object step by step from top to bottom; if the object corresponding to the original image is a shoe, the camera feature associated with this object type instructs the camera to display it around the target object. Alternatively, the terminal provides multiple preset camera features, and different preset camera features are used to indicate different trajectories and viewing angles of the camera. The terminal supports users to select a certain initial camera feature.

[0089] It should be noted that the terminal determines the camera features by the type of object corresponding to the original image, and can automatically match the camera features that are more suitable for the type of object, which is conducive to generating higher quality dynamic videos based on the camera features. The terminal provides a method for selecting preset camera features, which improves the flexibility of video generation and makes the video generation process more personalized. The above two methods enable the terminal to cope with a variety of application scenarios and needs, ensuring the effect and efficiency of video generation.

[0090] In some embodiments, the terminal extracts features from the original image through multiple modules in the first feature extraction model. Optionally, multiple modules can be processed simultaneously, and multiple feature modules can also be processed sequentially in a preset order, which is not limited here. Here, multiple feature modules extract features from the original image simultaneously as an example for explanation, see steps S302 to S304. It should be noted that the process of the terminal acquiring multiple features through the first feature extraction model and the process of the terminal acquiring camera features can be performed simultaneously, and can be processed sequentially in a preset order, which is not limited here. Here, the terminal first acquires camera features and then acquires multiple features through the first feature extraction model as an example for explanation.

[0091] In step S302, the terminal inputs the original image into the first module of the first feature extraction model to perform feature extraction to obtain semantic features. The first module is used to extract semantic information of the image input into the first module.

[0092] In the embodiments of the present disclosure, semantic features refer to high-level semantic information. The semantic features are usually used to identify the target object and the overall scene of the original image. Generally, semantic features are insufficient in indicating details such as texture and color. For the convenience of describing the overall model adopted by the video generation method, see Figure 4 As shown, Figure 4 4 is a schematic diagram of an overall model according to an exemplary embodiment. The terminal inputs the original image 401 into the first module for processing to obtain the semantic feature 402.

[0093] For example, the first module is a multimodal neural network that can learn from a large amount of image and text paired data to understand the semantic relationship between images and text and obtain semantic features. The semantic features enable different images to be compared in a semantic space.

[0094] In step S303, the terminal inputs the original image into the second module of the first feature extraction model to extract features and obtain implicit features. The implicit features are used to represent the potential information of the original image. The second module is used to obtain the potential information of the image input to the second module.

[0095] In the disclosed embodiment, the implicit feature refers to an image feature determined based on the key information of the input image. The implicit feature describes the original image at a resolution lower than that of the original image. Generally, the implicit feature can indicate most of the features in the original image, and the majority of the features include at least color features and texture features. Accordingly, the terminal encodes the original image through the second module to obtain low-resolution implicit features, thereby reducing the amount of calculation and improving the model processing efficiency. Figure 4 The terminal inputs the original image 401 into the second module for processing to obtain the latent feature 403. For example, the second module is a generative model for learning the distribution of data in the latent space.

[0096] In step S304, the terminal inputs the original image into the third module of the first feature extraction model for feature extraction to obtain masking features, which are used to distinguish the target object in the original image and the part of the original image other than the target object. The third module is used to segment the foreground and background in the image input to the third module.

[0097] In the disclosed embodiment, for the original image, the foreground is the target object, and the background is the non-target object, that is, the part other than the target object. The third module is used to determine the area where the target object and the area where the non-target object are located in the input image according to the foreground segmentation algorithm. Among them, the foreground segmentation algorithm distinguishes the foreground and background based on the characteristics of the pixel points in the image (such as color, texture, position, etc.), and separates the foreground object from the complex background. For example, in a commodity display scenario, the terminal distinguishes between commodity areas and non-commodity areas through the foreground segmentation algorithm, which facilitates the camera movement with the commodity as the axis center in the subsequent process to obtain a dynamic video that conforms to the display logic. Alternatively, the terminal distinguishes between the area where the model wearing the commodity is located and other areas except the model through the foreground segmentation algorithm, which facilitates the camera movement with the model as the axis center in the subsequent process to obtain a dynamic video that conforms to the display logic. See. Figure 4 As shown, the terminal obtains the mask feature 404 according to the original image 401 through the foreground segmentation algorithm. For example, the third module is any module that applies the foreground segmentation algorithm.

[0098] It should be noted that the terminal extracts various types of features through different modules, obtains richer information, and improves the diversity of feature extraction. In the subsequent process, the fusion of multiple features can provide a more comprehensive image description, which is conducive to the video generation model to learn the original image more comprehensively. At the same time, the modular design makes the image feature extraction process more flexible, facilitates subsequent model adjustment and expansion, and improves the flexibility and adaptability of the model. It should be noted that the first module, the second module, the third module and other modules can be regarded as separate models that do not affect each other, and will not be repeated here.

[0099] In step S305, the terminal copies the latent features, masked features and noise data respectively to obtain a target number of latent features, a target number of masked features and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated.

[0100] In the disclosed embodiment, when the target number has been determined, the target number of images is obtained by predicting the video image of each frame, and the dynamic video is obtained by splicing. The video image of each frame is generated based on multiple information, and the multiple information includes implicit features and masked features. Therefore, before the terminal obtains the video image through the video generation model, the terminal copies the implicit features, masked features, etc. that have been obtained, respectively, to obtain the implicit features of the target number and the masked features of the target number. The implicit features of the target number and the masked features of the target number correspond one-to-one to the preset number of video frames. In the above manner, the terminal ensures that the multiple information corresponding to the video frame can be obtained in a timely manner when generating each frame of the video image. See Figure 4 As shown, the terminal copies a single implicit feature 403 into a target number of implicit features 405 , and the terminal copies a single masked feature 404 into a target number of masked features 406 .

[0101] In step S306, the terminal performs feature fusion on the target quantity's hidden features, the target quantity's masked features, and the target quantity's noise data, respectively, on the hidden features corresponding to each video frame, the masked features corresponding to the video frame, and the noise data corresponding to the video frame to obtain the target quantity's intermediate features.

[0102] In the embodiment of the present disclosure, since the latent features and masked features are the direct input data of the video generation model, for the data corresponding to each frame of video image, the terminal concatenates the latent features, masked features and noise data in the channel dimension, thereby performing feature fusion to obtain intermediate features. Figure 4As shown, the terminal concatenates the target quantity's implicit features 406 , the target quantity's masked features 406 , and the target quantity's noise data 407 in the channel dimension to obtain the target quantity's intermediate features.

[0103] It should be noted that the terminal copies implicit features, masked features and noise data to the target number so that each frame of video corresponds to an independent feature representation, which not only increases the controllability of the video generation process, but also makes the video generation of each frame based on more specific and rich feature information, which is conducive to generating more coherent dynamic video, thereby improving the quality and usability of the generated dynamic video.

[0104] In step S307, the terminal uses the intermediate features as input data of the input layer of the first video generation model, uses the camera features and semantic features as data for feature fusion in each layer of the first video generation model, encodes and decodes based on the first video generation model, and obtains a dynamic video of the target object, which is used to display the target object based on the camera features.

[0105] In the disclosed embodiment, the first video generation model refers to a trained neural network model for the deduction process. The model is used to predict dynamic video based on the input image. The terminal uses the intermediate features as direct input data of the model, and the camera features and semantic features as conditional data of the model. Among them, the conditional data refers to information provided to the model in addition to the direct input data.

[0106] See also Figure 4 As shown in the figure, the left side of the model is the encoder part, which gradually reduces the size of the feature map through multiple convolutional layers and pooling layers; the right side is the decoder part, which gradually increases the size of the feature map through upsampling and convolutional layers; at the same time, the model has a timing module to process the temporal dependency between video images of different frames. The model generates dynamic videos containing the target number of video frame images through the above structure.

[0107] Optionally, the conditional data interacts with each layer of the video generation model by embedding to guide the data processing process and improve the model performance and data processing efficiency. For camera features, the terminal injects the camera features into the latent features by point addition before the timing module. The latent features are used to indicate the feature maps extracted from each layer of the model. In other words, the camera features and latent features are fused by adding the corresponding elements one by one, which facilitates the subsequent combination of camera movement and time information in the timing module to generate a dynamic video that is more in line with the display logic. See Figure 4 As shown, semantic features 402 and camera features 408 interact with each layer in the video generation model as conditional data. The model outputs dynamic video 409.

[0108] It should be noted that the terminal generates intermediate features based on implicit features and masked features combined with noise data, which can more accurately describe the contours and details of the target object, making the subsequently generated dynamic video more accurate and rich in details. At the same time, the terminal uses camera features and semantic features as data for feature fusion in the process of model-generated video, so that the model can adjust the video generation process and data according to the above features, making the video generation process more flexible. In other words, the terminal introduces the above-mentioned multiple features into the video generation model and guides the video generation model to perform denoising, so that the model can predict a dynamic video with high consistency of the target object, rich and accurate details, and reasonable motion trajectory and motion amplitude. The above method improves the efficiency of video generation and is conducive to generating more accurate and higher-quality dynamic videos.

[0109] In some embodiments, the training steps of the first feature extraction model and the first video generation model include the following steps 1 to 2.

[0110] Step 1: Obtain multiple training data sets, the multiple training data sets include a first data set, a second data set and a third data set, the first data set includes multiple sample videos, the second data set includes sample videos of sample objects in a target field, the target field is the field to which the target object belongs, and the third data set includes sample videos in the second data set that meet preset conditions.

[0111] For ease of description, the first data set may also be referred to as a general data set. The sample objects in the first data set are common objects in life and are divided into multiple fields. For example, the sample objects in the first data set include butterflies, shirts, lawns, water cups, etc. Optionally, the target field is the e-commerce field. The sample objects in the second data set are all objects in the e-commerce field. For example, the sample objects in the second data set include commodities such as shirts and water cups, but do not include natural objects such as butterflies and lawns in nature. The third data set includes high-quality data in the second data set. The sample videos in the third data set have the characteristics of high definition, high aesthetics, and advanced camera movements.

[0112] Step 2: Based on the first data set, the second data set and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model.

[0113] In some embodiments, the terminal performs multi-stage training on the model by selecting different data sets, as shown in (1)-(3) below.

[0114] (1) The terminal performs a first-stage training on a second feature extraction model and a second video generation model based on the first data set and the second data set to obtain a first-stage feature extraction model and a first-stage video generation model. The dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set.

[0115] The second feature extraction model and the second video generation model are both trained models. The terminal is trained with the first data set and the second data set, so that the model can learn the motion laws of common sample objects and the motion laws of the camera, thereby enhancing the generalization ability of the model.

[0116] (2) Based on the second data set, the terminal performs a second-stage training on the feature extraction model of the first stage and the video generation model of the first stage to obtain a second-stage feature extraction model and a second-stage video generation model. The second-stage video generation model is used to generate dynamic video corresponding to the target domain.

[0117] The terminal fine-tunes the model through the second data set, so that the model can learn the movement rules of sample objects in the target field, and then generate dynamic videos that are more in line with the characteristics of the target field, thereby improving the quality and relevance of the generated video.

[0118] (3) Based on the third data set, the terminal performs a third-stage training on the feature extraction model of the second stage and the video generation model of the second stage to obtain a first feature extraction model and a first video generation model. The video score of the dynamic video output by the video generation model of the third stage is higher than the video score of the dynamic video output by the video generation model of the second stage.

[0119] The terminal further fine-tunes the model through the third data set, so that the model can learn the camera movement rules of a better camera, which is conducive to further optimizing the effect and quality of the generated video.

[0120] It should be noted that the terminal gradually trains in the first, second and third stages, so that the model gradually learns the information of different data sets in depth, that is, gradually from generalized learning to in-depth learning in specific fields, and then fine-tunes and optimizes the model, which improves the overall training effect and quality. At the same time, since each stage is optimized for a specific training goal, it is beneficial for the model to better understand and adapt to the training goals of different stages, so it can achieve the expected training effect faster and improve the efficiency of model training.

[0121] In some embodiments, for any training stage, the terminal updates the model parameters according to the predicted video and the sample video. Accordingly, the terminal inputs the sample video in the training data set into the current video processing model for feature extraction to obtain the sample camera feature, which is used to represent the trajectory and viewing angle of the camera of the sample video; the terminal inputs the first frame of the sample video into the current feature extraction model for feature extraction to obtain the sample semantic feature, the sample implicit feature and the sample masking feature, which are used to represent the potential information of the first frame of the video image, and the sample masking feature is used to distinguish the sample object in the first frame of the video image and the part other than the sample object in the first frame of the video image; the terminal inputs the sample semantic feature, the sample implicit feature, the sample masking feature and the sample camera feature into the current video generation model for decoding and encoding to obtain the predicted video, which is used to display the sample object based on the sample camera feature; the terminal updates the parameters of the current feature extraction model, the current video generation model and the current video processing model through the objective function based on the error between the predicted video and the sample video, and obtains the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

[0122] It should be noted that in the video generation method of the embodiment of the present disclosure, in addition to inputting camera features into the video generation model for training, sample videos can also be directly input into the video generation model for training, which will not be elaborated here.

[0123] Among them, the video processing model is used to extract camera features. Optionally, during the training process, the terminal uses a feature extraction tool to extract information from the sample video, which contains at least a 1x3 translation vector and a 3x3 rotation matrix. After flattening the information, the feature extraction network model is used to extract camera features. Among them, the translation vector is used to represent the linear movement distance of the camera along the X, Y, and Z axes in three-dimensional space. The rotation matrix is ​​used to describe the rotation direction and rotation angle of the camera in three-dimensional space. The rotation matrix is ​​also used to convert a point in the camera coordinate system to another coordinate system, which will not be repeated here. Therefore, the camera feature describes the way the camera moves in three-dimensional space when shooting a video, that is, the way the camera moves.

[0124] It should be noted that, in the video generation process, the subsequent video frames I1, I2, ..., I N, thereby obtaining a dynamic video. Among them, the initial reference frame of the target object is the original image, and is also the first frame of video image in the dynamic video. The video generation method in the disclosed embodiment adopts a diffusion model, and the objective function of the model is used to indicate the difference between the predicted noise and the real noise of the model. Among them, the predicted noise of the model is determined by the time step, implicit features, initial reference frame and camera features. The time step is used to represent the process of the diffusion process from the initial state to the noisy state and from the noisy state to the clear state. Take the expected values ​​of the implicit features, initial reference frame, camera features, time step and noise data to obtain the total loss value corresponding to the model. That is, compared with the objective function of the traditional diffusion model, the objective function of the model in the disclosed embodiment also includes camera features.

[0125] It should be noted that in any training stage, the terminal simultaneously extracts the camera features of the sample video and the semantic features, implicit features, and masked features of the first frame of the video image, so that the model can obtain richer and more comprehensive information, which is conducive to maintaining the consistency of the video generation process. At the same time, by introducing camera features during the training process, the terminal enables the model to simulate the movement and behavior of the camera in the display scene of the target object, thereby improving the quality and usability of the generated video. In addition, the terminal evaluates the model performance by predicting the error between the video and the sample video, and continuously adjusts and optimizes the model parameters to improve the model performance.

[0126] The disclosed embodiment provides a video generation method, in which multiple image features and camera features such as semantic features, implicit features, and masked features extracted from the original image are used as inputs to a first video generation model, so that the first video generation model can not only display the target object in the original image in the dynamic video, but also learn the camera features, so that the camera trajectory and camera perspective of the dynamic video are more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces multiple image features and camera features, so that the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, and improves the video quality and availability.

[0127] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0128] Figure 5 FIG. 1 is a block diagram of a video generating device according to an exemplary embodiment. Figure 5 As shown, the device includes: a determining unit 501, an extracting unit 502 and a generating unit 503.

[0129] A determination unit 501 is configured to obtain a camera feature associated with an original image, the original image includes a target object, and the camera feature is used to represent a trajectory and a viewing angle of the camera;

[0130] The extraction unit 502 is configured to input the original image into the first feature extraction model to perform feature extraction, and obtain semantic features, latent features and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from the part of the original image other than the target object;

[0131] The generation unit 503 is configured to input the semantic features, implicit features, masked features and camera features into the first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

[0132] In some embodiments, the determination unit 501 is configured to determine the camera feature associated with the item type as the camera feature associated with the original image based on the item type corresponding to the original image, where the item type is the type of the target object contained in the original image; or, in a case where multiple preset camera features have been set, in response to a selection operation on any preset camera feature, determine the preset camera feature as the camera feature associated with the original image, where different preset camera features are used to represent different camera trajectories and viewing angles.

[0133] In some embodiments, Figure 6 is a block diagram of another video generating device according to an exemplary embodiment. Figure 6 As shown, the generating unit 503 includes:

[0134] The fusion subunit 5031 is configured to perform feature fusion on the latent features, the masked features and the noise data to obtain the intermediate features;

[0135] The generation subunit 5032 is configured to use the intermediate features as input data of the input layer of the first video generation model, use the camera features and semantic features as data for feature fusion in each layer of the first video generation model, encode and decode based on the first video generation model, and obtain a dynamic video of the target object.

[0136] In some embodiments, the fusion subunit 5031 is configured to copy the latent features, the masked features and the noise data respectively to obtain a target number of latent features, a target number of masked features and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated; for the target number of latent features, the target number of masked features and the target number of noise data, the target number of intermediate features are obtained for the latent features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame.

[0137] In some embodiments, the apparatus further comprises:

[0138] The acquisition unit 601 is configured to acquire multiple training data sets, the multiple training data sets include a first data set, a second data set and a third data set, the first data set includes multiple sample videos, the second data set includes sample videos of sample objects in a target domain, the target domain is the domain to which the target object belongs, and the third data set includes sample videos in the second data set that meet a preset condition;

[0139] The training unit 602 is configured to train the second feature extraction model and the second video generation model based on the first data set, the second data set and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

[0140] In some embodiments, the training unit 602 is configured to perform a first-stage training on the second feature extraction model and the second video generation model based on the first data set and the second data set to obtain a first-stage feature extraction model and a first-stage video generation model, and the dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set; based on the second data set, perform a second-stage training on the first-stage feature extraction model and the first-stage video generation model to obtain a second-stage feature extraction model and a second-stage video generation model, and the second-stage video generation model is used to generate dynamic video corresponding to the target domain; based on the third data set, perform a third-stage training on the second-stage feature extraction model and the second-stage video generation model to obtain a first feature extraction model and a first video generation model, and the video score of the dynamic video output by the third-stage video generation model is higher than the video score of the dynamic video output by the second-stage video generation model.

[0141] In some embodiments, the training unit 602 is configured to, for any training stage, input the sample video in the training data set into the current video processing model for feature extraction to obtain sample camera features, where the sample camera features are used to represent the trajectory and viewing angle of the camera of the sample video; input the first frame of the sample video image into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features and sample masking features, where the sample latent features are used to represent the potential information of the first frame of the video image, and the sample masking features are used to distinguish the sample object in the first frame of the video image and the part of the first frame of the video image other than the sample object; input the sample semantic features, sample latent features, sample masking features and sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, where the predicted video is used to display the sample object based on the sample camera features; based on the error between the predicted video and the sample video, update the parameters of the current feature extraction model, the current video generation model and the current video processing model through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

[0142] The disclosed embodiment provides a video generation device, in which the terminal uses multiple image features and camera features such as semantic features, implicit features, and masked features extracted from the original image as inputs to the first video generation model, so that the first video generation model can not only display the target object in the original image in the dynamic video, but also learn the camera features, so that the camera trajectory and camera perspective of the dynamic video are more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, the present device introduces multiple image features and camera features, so that the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, and improves the video quality and availability.

[0143] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

[0144] It should be noted that the video generation device provided in the above embodiment is only illustrated by the division of the above functional units. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the video generation device provided in the above embodiment and the video generation method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0145] Regarding the video generating device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] Figure 7 7 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 700 includes: a processor 701 and a memory 702 .

[0147] The processor 701 may include one or more processing cores, such as a 4-core processor, a 7-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0148] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the video generation method provided by the method embodiment of the present disclosure.

[0149] In some embodiments, the electronic device 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707 and a power supply 708.

[0150] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0151] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0152] The display screen 705 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, and the front panel of the electronic device 700 is set; in other embodiments, the display screen 705 can be at least two, which are respectively set on different surfaces of the electronic device 700 or are folded; in some other embodiments, the display screen 705 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0153] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0154] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0155] The power supply 708 is used to power various components in the electronic device 700. The power supply 708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0156] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the electronic device 700, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0157] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions, and the instructions can be executed by a processor 701 of an electronic device 700 to complete the above-mentioned video generation method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] A computer program product comprises a computer program, and when the computer program is executed by a processor, the video generating method is implemented.

[0159] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0160] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a camera feature associated with an original image, wherein the original image includes a target object, and the camera feature is used to represent a trajectory and a viewing angle of the camera; Inputting the original image into a first feature extraction model for feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from the portion of the original image other than the target object; The semantic features, the latent features, the masked features and the camera features are input into a first video generation model for decoding and encoding to obtain a dynamic video, where the dynamic video is used to display the target object based on the camera features.

2. The video generation method according to claim 1, characterized in that: The acquiring of camera features associated with the original image comprises: Based on the object type corresponding to the original image, determining the camera feature associated with the object type as the camera feature associated with the original image, the object type being the type of the target object contained in the original image; or, In the case where multiple preset camera features are set, in response to a selection operation on any preset camera feature, the preset camera feature is determined as the camera feature associated with the original image, and different preset camera features are used to represent different camera trajectories and viewing angles.

3. The video generation method according to claim 1, characterized in that: The step of inputting the semantic feature, the implicit feature, the mask feature, and the camera feature into a first video generation model for decoding and encoding includes: Performing feature fusion on the latent feature, the masked feature and the noise data to obtain an intermediate feature; The intermediate features are used as input data of the input layer of the first video generation model, the camera features and the semantic features are used as data for feature fusion in each layer of the first video generation model, encoding and decoding are performed based on the first video generation model to obtain a dynamic video of the target object.

4. The video generation method according to claim 3, characterized in that: The step of fusing the latent feature, the masked feature and the noise data to obtain an intermediate feature includes: The latent features, the masked features, and the noise data are respectively copied to obtain a target number of latent features, a target number of masked features, and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated; For the implicit features of the target number, the masked features of the target number and the noise data of the target number, feature fusion is performed on the implicit features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame to obtain the intermediate features of the target number.

5. The video generation method according to claim 1, characterized in that: The training steps of the first feature extraction model and the first video generation model include: Acquire multiple training data sets, the multiple training data sets include a first data set, a second data set and a third data set, the first data set includes multiple sample videos, the second data set includes sample videos of sample objects in a target field, the target field is the field to which the target objects belong, and the third data set includes sample videos in the second data set that meet preset conditions; Based on the first data set, the second data set and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

6. The video generation method according to claim 5, characterized in that: The method of training a second feature extraction model and a second video generation model based on the first data set, the second data set, and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model includes: Based on the first data set and the second data set, the second feature extraction model and the second video generation model are trained in the first stage to obtain a first-stage feature extraction model and a first-stage video generation model, wherein the dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set; Based on the second data set, performing a second-stage training on the feature extraction model of the first stage and the video generation model of the first stage to obtain a second-stage feature extraction model and a second-stage video generation model, wherein the second-stage video generation model is used to generate a dynamic video corresponding to the target domain; Based on the third data set, the feature extraction model of the second stage and the video generation model of the second stage are trained for the third stage to obtain the first feature extraction model and the first video generation model, and the video score of the dynamic video output by the video generation model of the third stage is higher than the video score of the dynamic video output by the video generation model of the second stage.

7. The video generation method according to claim 5, characterized in that: The method of training a second feature extraction model and a second video generation model based on the first data set, the second data set, and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model includes: For any training stage, the sample video in the training data set is input into the current video processing model for feature extraction to obtain sample camera features, where the sample camera features are used to represent the trajectory and viewing angle of the camera of the sample video; Inputting a first frame of the sample video into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features, and sample masking features, wherein the sample latent features are used to represent potential information of the first frame of the video image, and the sample masking features are used to distinguish a sample object in the first frame of the video image and a portion of the first frame of the video image other than the sample object; Inputting the sample semantic features, the sample implicit features, the sample masked features and the sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, wherein the predicted video is used to display the sample object based on the sample camera features; Based on the error between the predicted video and the sample video, the parameters of the current feature extraction model, the current video generation model and the current video processing model are updated through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

8. A video generating device, characterized in that: The device comprises: a determining unit configured to obtain a camera feature associated with an original image, wherein the original image includes a target object, and wherein the camera feature is used to represent a trajectory and a viewing angle of a camera; an extraction unit configured to input the original image into a first feature extraction model to perform feature extraction, and obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from a portion of the original image other than the target object; The generation unit is configured to input the semantic features, the latent features, the masked features and the camera features into a first video generation model for decoding and encoding to obtain a dynamic video, wherein the dynamic video is used to display the target object based on the camera features.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the video generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video generating method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.

Images