Video generation method, device, electronic device and storage medium

Abstract

The present disclosure provides a video generation method, device, electronic device and storage medium, which belong to the field of computer technology. In this method, a variety of image features and camera features such as semantic features, implicit features, and masking features extracted from the original image are used as inputs of a first video generation model, so that the first video generation model can not only display the target object in the original image in the dynamic video, but also enable the first video generation model to learn the camera features, so that the camera trajectory and camera perspective of the dynamic video are more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces a variety of image features and camera features, so that the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, thereby improving the video quality and availability.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video generation method, device, electronic device, and storage medium. Background Art

[0002] In the context of item display, using dynamic videos is more effective at capturing user attention and eye contact than simply displaying items through static images. Dynamic videos can also better convey item information to users. However, when faced with a large number of items, batch-producing dynamic videos of these objects through filming is not only difficult but also inefficient. Therefore, how to quickly produce dynamic videos of objects is a problem that needs to be solved.

[0003] At present, the commonly used approach is to use the semantic features of the original image to guide the processing of the neural network model when there is an original image of the object, so that the neural network model generates a dynamic video associated with the semantic features.

[0004] However, because current neural network models tend to simulate the natural movement of objects in the real world, they often struggle to generate dynamic videos that align with the logical display of objects. This results in poor video generation, and the usability of the dynamic videos generated by neural network models is low. Summary of the Invention

[0005] The present disclosure provides a video generation method, device, electronic device, and storage medium. By introducing multiple image features and camera features, the first video generation model uses the target object as the main body to generate dynamic videos with richer details and more consistent with display logic, thereby improving video quality and usability. The technical solutions of the present disclosure are as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a video generation method is provided, the method comprising:

[0007] Obtaining camera features associated with an original image, wherein the original image includes a target object, the camera features being used to represent a trajectory and a viewing angle of the camera;

[0008] Inputting the original image into a first feature extraction model for feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from parts of the original image other than the target object;

[0009] The semantic features, the latent features, the masked features, and the camera features are input into a first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

[0010] According to another aspect of an embodiment of the present disclosure, a video generating apparatus is provided, the apparatus comprising:

[0011] a determining unit configured to obtain camera features associated with an original image, wherein the original image includes a target object, the camera features being used to represent a trajectory and a viewing angle of the camera;

[0012] an extraction unit configured to input the original image into a first feature extraction model to perform feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from a portion of the original image other than the target object;

[0013] The generation unit is configured to input the semantic features, the implicit features, the masked features and the camera features into a first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

[0014] In some embodiments, the determination unit is configured to determine the camera feature associated with the item type as the camera feature associated with the original image based on the item type corresponding to the original image, where the item type is the type of the target object contained in the original image; or, in a case where multiple preset camera features have been set, in response to a selection operation of any preset camera feature, determine the preset camera feature as the camera feature associated with the original image, where different preset camera features are used to represent different camera trajectories and viewing angles.

[0015] In some embodiments, the generating unit includes:

[0016] a fusion subunit, configured to perform feature fusion on the latent feature, the masked feature, and the noise data to obtain an intermediate feature;

[0017] The generation subunit is configured to use the intermediate features as input data of the input layer of the first video generation model, use the camera features and the semantic features as data for feature fusion in each layer of the first video generation model, and perform encoding and decoding based on the first video generation model to obtain a dynamic video of the target object.

[0018] In some embodiments, the fusion subunit is configured to copy the latent features, the masked features and the noise data respectively to obtain a target number of latent features, the target number of masked features and the target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated; for the target number of latent features, the target number of masked features and the target number of noise data, the target number of intermediate features are obtained for the latent features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame respectively.

[0019] In some embodiments, the apparatus further comprises:

[0020] an acquisition unit configured to acquire a plurality of training data sets, the plurality of training data sets including a first data set, a second data set, and a third data set, the first data set including a plurality of sample videos, the second data set including sample videos of sample objects in a target domain, the target domain being the domain to which the target objects belong, and the third data set including sample videos in the second data set that meet a preset condition;

[0021] The training unit is configured to train the second feature extraction model and the second video generation model based on the first data set, the second data set and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

[0022] In some embodiments, the training unit is configured to perform a first-stage training on the second feature extraction model and the second video generation model based on the first data set and the second data set to obtain a first-stage feature extraction model and a first-stage video generation model, and the dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set; based on the second data set, perform a second-stage training on the first-stage feature extraction model and the first-stage video generation model to obtain a second-stage feature extraction model and a second-stage video generation model, and the second-stage video generation model is used to generate dynamic video corresponding to the target field; based on the third data set, perform a third-stage training on the second-stage feature extraction model and the second-stage video generation model to obtain the first feature extraction model and the first video generation model, and the video score of the dynamic video output by the third-stage video generation model is higher than the video score of the dynamic video output by the second-stage video generation model.

[0023] In some embodiments, the training unit is configured to, for any training stage, input the sample video in the training data set into the current video processing model for feature extraction to obtain sample camera features, where the sample camera features are used to represent the trajectory and viewing angle of the camera of the sample video; input the first frame video image of the sample video into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features, and sample masking features, where the sample latent features are used to represent the potential information of the first frame video image, and the sample masking features are used to distinguish the sample object in the first frame video image and the part of the first frame video image other than the sample object; input the sample semantic features, the sample latent features, the sample masking features, and the sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, where the predicted video is used to display the sample object based on the sample camera features; based on the error between the predicted video and the sample video, update the parameters of the current feature extraction model, the current video generation model, and the current video processing model through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage, and the video processing model of the training stage.

[0024] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, the electronic device including:

[0025] one or more processors;

[0026] a memory for storing program codes executable by the processor;

[0027] The processor is configured to execute the program code to implement the above-mentioned video generation method.

[0028] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned video generating method.

[0029] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned video generation method when executed by a processor.

[0030] The disclosed embodiment provides a video generation method, in which multiple image features and camera features, such as semantic features, latent features, and masked features, extracted from an original image are used as input to a first video generation model, so that the first video generation model can not only display the target object in the original image in a dynamic video, but also learn the camera features, thereby making the camera trajectory and camera perspective of the dynamic video more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces multiple image features and camera features, so that the first video generation model uses the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, thereby improving the video quality and availability.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0033] Figure 1 The figure is a schematic diagram showing an implementation environment of a video generation method according to an exemplary embodiment.

[0034] Figure 2 The figure is a flowchart of a video generating method according to an exemplary embodiment.

[0035] Figure 3 The figure is a flowchart of another method for generating a video according to an exemplary embodiment.

[0036] Figure 4 It is a schematic diagram of an overall model according to an exemplary embodiment.

[0037] Figure 5 The figure is a block diagram of a video generating apparatus according to an exemplary embodiment.

[0038] Figure 6 is a block diagram showing another video generating apparatus according to an exemplary embodiment.

[0039] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0041] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0042] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, and display, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the original images involved in this disclosure were all obtained with full authorization.

[0043] Figure 1 FIG. 1 is a schematic diagram showing an implementation environment of a video generation method according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0044] Terminal 101 may be a device such as a smartphone, smartwatch, desktop computer, laptop, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), or laptop computer. Terminal 101 may have an application installed and running that generates a dynamic video from an original image. This application is associated with server 102, which provides backend services to terminal 101.

[0045] Terminal 101 may generally refer to one of multiple terminals. This embodiment uses terminal 101 as an example. Those skilled in the art will appreciate that the number of terminals may be greater or lesser. For example, there may be a few terminals, or dozens, hundreds, or even more. This embodiment does not limit the number or device type of terminals.

[0046] The server 102 is a server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, and the embodiments of the present disclosure do not limit this. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, and the embodiments of the present disclosure do not limit this.

[0047] Figure 2 is a flow chart of a video generation method according to an exemplary embodiment. Figure 2 As shown, the method is executed by the terminal and includes the following steps:

[0048] In step S201, the terminal obtains camera features associated with an original image, where the original image includes a target object. The camera features are used to represent a trajectory and a viewing angle of the camera.

[0049] In the disclosed embodiments, the original image includes a target object, and information such as the target object's color, shape, and form can be obtained from the original image. For example, if the target object is a clothing item, the original image is a reference image of the clothing item. For ease of description, camera features can also be referred to as camera movement features. Different camera features correspond to different camera trajectories and camera perspectives.

[0050] In step S202, the terminal inputs the original image into the first feature extraction model for feature extraction to obtain semantic features, latent features and masked features. The latent features are used to represent the potential information of the original image, and the masked features are used to distinguish the target object in the original image and the part of the original image other than the target object.

[0051] In the disclosed embodiments, the first feature extraction model refers to a trained model for predicting dynamic video. Optionally, the model includes multiple modules, each of which is used to extract features of different dimensions or levels based on the original image. When the terminal extracts features from the original image using multiple modules in the model, the multiple modules can optionally be processed simultaneously, or the multiple feature modules can be processed sequentially in a preset order, which is not limited here.

[0052] Hidden features are image features determined based on key information from the input image. These features describe the original image at a lower resolution than the original image. Masked features are used to distinguish target and non-target areas in the original image by using different pixel values.

[0053] In step S203, the terminal inputs the semantic features, implicit features, masked features, and camera features into the first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

[0054] In the disclosed embodiment, the first video generation model generates multiple frames of video images based on various feature information and camera features in the original image, and then generates a dynamic video based on the multiple frames of video images. The camera movement of the dynamic video refers to the camera movement indicated by the camera features learned by the model.

[0055] The disclosed embodiment provides a video generation method, in which multiple image features and camera features, such as semantic features, latent features, and masked features, extracted from an original image are used as input to a first video generation model, so that the first video generation model can not only display the target object in the original image in a dynamic video, but also learn the camera features, thereby making the camera trajectory and camera perspective of the dynamic video more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces multiple image features and camera features, so that the first video generation model uses the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, thereby improving the video quality and availability.

[0056] In some embodiments, obtaining camera features associated with the original image includes:

[0057] Based on the item type corresponding to the original image, the camera features associated with the item type are determined as the camera features associated with the original image, where the item type is the type of the target object contained in the original image; or

[0058] In the case where multiple preset camera features are set, in response to a selection operation on any preset camera feature, the preset camera feature is determined as the camera feature associated with the original image, and different preset camera features are used to represent different camera trajectories and viewing angles.

[0059] In the disclosed embodiments, the terminal determines camera features based on the object type corresponding to the original image, automatically matching camera features more appropriate for that type of object. This facilitates the generation of higher-quality dynamic videos based on these camera features. Furthermore, the terminal provides a method for selecting preset camera features, increasing the flexibility of video generation and making the video generation process more personalized. These two methods enable the terminal to address a variety of application scenarios and needs, ensuring the effectiveness and efficiency of video generation.

[0060] In some embodiments, inputting semantic features, latent features, masked features, and camera features into a first video generation model for decoding and encoding includes:

[0061] Fuse the latent features, masked features and noise data to obtain intermediate features;

[0062] The intermediate features are used as input data of the input layer of the first video generation model, the camera features and semantic features are used as data for feature fusion in each layer of the first video generation model, and encoding and decoding are performed based on the first video generation model to obtain a dynamic video of the target object.

[0063] In the disclosed embodiment, the terminal generates intermediate features based on implicit features and masked features in combination with noise data, which can more accurately describe the outline and details of the target object, making the subsequently generated dynamic video more accurate and rich in details. At the same time, the terminal uses camera features and semantic features as data for feature fusion in the process of model-generated video, so that the model can adjust the video generation process and data according to the above features, making the video generation process more flexible. The above method improves the efficiency of video generation and is conducive to generating more accurate and higher-quality dynamic videos.

[0064] In some embodiments, latent features, masked features, and noise data are fused to obtain intermediate features, including:

[0065] The latent features, the masked features, and the noise data are respectively copied to obtain a target number of latent features, a target number of masked features, and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated;

[0066] For the target number of hidden features, the target number of masked features and the target number of noise data, the target number of intermediate features are obtained for the hidden features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame respectively.

[0067] In the disclosed embodiment, the terminal copies implicit features, masked features, and noise data to a target number so that each frame of video corresponds to an independent feature representation. This not only increases the controllability of the video generation process, but also enables the video generation of each frame to be based on more specific and rich feature information, which is conducive to generating more coherent dynamic video, thereby improving the quality and usability of the generated dynamic video.

[0068] In some embodiments, the steps of training the first feature extraction model and the first video generation model include:

[0069] Acquire multiple training data sets, the multiple training data sets include a first data set, a second data set, and a third data set, the first data set includes multiple sample videos, the second data set includes sample videos of sample objects in a target domain, where the target domain is the domain to which the target objects belong, and the third data set includes sample videos in the second data set that meet preset conditions;

[0070] Based on the first data set, the second data set and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

[0071] In the embodiment of the present disclosure, the terminal performs model training through the first data set, so that the model can learn common camera features during the training process, thereby enhancing the generalization ability of the model. Then, the terminal fine-tunes the model through the second data set, so that the model can generate dynamic videos that are more in line with the characteristics of the target field, thereby improving the quality and relevance of the generated video. The terminal further fine-tunes the model through the third data set, so that the model can learn higher-quality camera features, which is conducive to further optimizing the effect and quality of the generated video. Through multi-stage training, the terminal enables the model to gradually and deeply learn the information of different data sets, thereby improving the overall training effect and quality. At the same time, since each stage is optimized for a specific training goal, the expected training effect can be achieved more quickly, thereby improving the efficiency of model training.

[0072] In some embodiments, based on the first data set, the second data set, and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, including:

[0073] Based on the first data set and the second data set, a first-stage training is performed on the second feature extraction model and the second video generation model to obtain a first-stage feature extraction model and a first-stage video generation model, wherein the first-stage video generation model outputs a dynamic video that reflects the camera features indicated by the first data set and the camera features indicated by the second data set;

[0074] Based on the second data set, the feature extraction model of the first stage and the video generation model of the first stage are trained in the second stage to obtain the feature extraction model of the second stage and the video generation model of the second stage. The video generation model of the second stage is used to generate dynamic videos corresponding to the target domain;

[0075] Based on the third data set, the feature extraction model of the second stage and the video generation model of the second stage are trained in the third stage to obtain the first feature extraction model and the first video generation model. The video score of the dynamic video output by the video generation model of the third stage is higher than the video score of the dynamic video output by the video generation model of the second stage.

[0076] In the disclosed embodiment, the terminal gradually trains the model through the first, second, and third stages, enabling it to gradually transition from generalized learning to in-depth learning in specific areas, followed by model fine-tuning and optimization. This approach helps the model better understand and adapt to the training objectives at different stages, improving not only the overall training effect and quality but also the efficiency of model training.

[0077] In some embodiments, based on the first data set, the second data set, and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, including:

[0078] For any training stage, the sample video in the training dataset is input into the current video processing model for feature extraction to obtain sample camera features. The sample camera features are used to represent the trajectory and viewing angle of the camera of the sample video.

[0079] Inputting the first frame of the sample video into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features, and sample masking features. The sample latent features are used to represent the potential information of the first frame of the video image, and the sample masking features are used to distinguish the sample object in the first frame of the video image from the portion of the first frame of the video image other than the sample object.

[0080] Input the sample semantic features, sample latent features, sample masking features, and sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, which is used to display the sample object based on the sample camera features;

[0081] Based on the error between the predicted video and the sample video, the parameters of the current feature extraction model, the current video generation model and the current video processing model are updated through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

[0082] In the embodiment of the present disclosure, at any training stage, the terminal simultaneously extracts the camera features of the sample video and the semantic features, implicit features, and masked features of the first frame of the video image, so that the model can obtain richer and more comprehensive information, which is conducive to maintaining the consistency of the video generation process. At the same time, by introducing camera features during the training process, the terminal enables the model to simulate the movement and behavior of the camera in the display scene of the target object, thereby improving the quality and usability of the generated video. In addition, the terminal evaluates the model performance by predicting the error between the video and the sample video, and continuously adjusts and optimizes the model parameters to improve the model performance.

[0083] above Figure 2 The figure shows a flow chart of a video generation method of the present disclosure. The video generation solution provided by the present disclosure is further described below. Figure 3 is a flowchart of another video generation method according to an exemplary embodiment. Figure 3 The method is executed by the terminal. The method includes the following steps:

[0084] In step S301, the terminal obtains camera features associated with an original image, where the original image includes a target object. The camera features are used to represent a trajectory and a viewing angle of the camera.

[0085] In the disclosed embodiments, the original image includes the target object, and information such as the target object's color, shape, and form can be obtained from the original image. For example, in a product display scenario, the target object may be an item such as fruit, household items, or clothing items. Alternatively, for clothing items, the target object may include only the clothing item or a model wearing the clothing item, without limitation. Accordingly, the original image refers to a reference product image that includes the target object.

[0086] For ease of description, the camera features associated with the original image are also referred to as camera movement features. When a dynamic video of a target object is generated using different camera features, the camera movement of the target object in the dynamic video will not be exactly the same, manifested as different camera trajectories and camera perspectives.

[0087] In some embodiments, the terminal obtains the camera features associated with the original image through various methods. Accordingly, based on the type of item corresponding to the original image, the terminal determines the camera features associated with the item type as the camera features associated with the original image, where the item type is the type of target object contained in the original image. Alternatively, if multiple preset camera features have been set, the terminal, in response to a selection of any preset camera feature, determines the preset camera feature as the camera feature associated with the original image, where different preset camera features are used to represent different camera trajectories and viewing angles.

[0088] For example, if the original image corresponds to a large appliance, the camera feature associated with this object type instructs the camera to display the target object from top to bottom. Alternatively, if the original image corresponds to a shoe, the camera feature associated with this object type instructs the camera to display the target object in a circular motion. Alternatively, the terminal provides multiple preset camera features, each of which indicates different camera trajectories and viewing angles. The terminal allows users to select a specific initial camera feature.

[0089] It's important to note that the terminal determines camera features based on the object type corresponding to the original image, automatically matching camera features more appropriate for that object type. This facilitates generating higher-quality dynamic videos based on these camera features. The terminal also offers a method for selecting preset camera features, increasing the flexibility of video generation and making the process more personalized. These two methods enable the terminal to address a variety of application scenarios and needs, ensuring both effective and efficient video generation.

[0090] In some embodiments, the terminal extracts features from the original image through multiple modules in the first feature extraction model. Optionally, multiple modules can be processed simultaneously, and multiple feature modules can also be processed in sequence according to a preset order, which is not limited here. Here, multiple feature modules extract features from the original image at the same time as an example for explanation, see steps S302 to S304. It should be noted that the process of the terminal obtaining multiple features through the first feature extraction model and the process of the terminal obtaining camera features can be carried out simultaneously, and can be processed in sequence according to a preset order, which is not limited here. Here, the terminal first obtains camera features and then obtains multiple features through the first feature extraction model as an example for explanation.

[0091] In step S302, the terminal inputs the original image into the first module of the first feature extraction model to perform feature extraction to obtain semantic features. The first module is used to extract semantic information of the image input into the first module.

[0092] In the embodiments of the present disclosure, semantic features refer to high-level semantic information. These semantic features are usually used to identify the target object and the overall scene of the original image. Generally, semantic features are insufficient in indicating details such as texture and color. To facilitate the description of the overall model used in the video generation method, see Figure 4 As shown, Figure 4 FIG. 4 is a schematic diagram of an overall model according to an exemplary embodiment. The terminal inputs an original image 401 into the first module for processing to obtain a semantic feature 402 .

[0093] For example, the first module is a multimodal neural network that can learn from a large amount of image and text paired data to understand the semantic relationship between images and text and obtain semantic features. These semantic features enable different images to be compared in a semantic space.

[0094] In step S303, the terminal inputs the original image into the second module of the first feature extraction model for feature extraction to obtain implicit features. The implicit features are used to represent the potential information of the original image. The second module is used to obtain the potential information of the image input to the second module.

[0095] In the embodiments of the present disclosure, implicit features refer to image features determined based on key information of the input image. The implicit features describe the original image at a lower resolution than the original image. Generally, the implicit features can indicate most features in the original image, which include at least color features and texture features. Accordingly, the terminal encodes the original image through the second module to obtain low-resolution implicit features, thereby reducing the amount of calculation and improving the model processing efficiency. Figure 4 The terminal inputs the original image 401 into the second module for processing to obtain latent features 403. For example, the second module is a generative model for learning the distribution of data in a latent space.

[0096] In step S304, the terminal inputs the original image into the third module of the first feature extraction model for feature extraction to obtain masking features. The masking features are used to distinguish the target object in the original image and the part of the original image other than the target object. The third module is used to segment the foreground and background in the image input to the third module.

[0097] In the embodiment of the present disclosure, for the original image, the foreground is the target object, and the background is the non-target object, that is, the part other than the target object. The third module is used to determine the area where the target object is located and the area where the non-target object is located in the input image according to the foreground segmentation algorithm. Among them, the foreground segmentation algorithm distinguishes the foreground and background based on the characteristics of the pixel points in the image (such as color, texture, position, etc.), and separates the foreground object from the complex background. For example, in a commodity display scenario, the terminal distinguishes between commodity areas and non-commodity areas through the foreground segmentation algorithm, which facilitates the camera movement with the commodity as the axis center in the subsequent process to obtain a dynamic video that conforms to the display logic. Alternatively, the terminal distinguishes between the area where the model wearing the commodity is located and other areas other than the model through the foreground segmentation algorithm, which facilitates the camera movement with the model as the axis center in the subsequent process to obtain a dynamic video that conforms to the display logic. See Figure 4 As shown, the terminal obtains the mask feature 404 according to the original image 401 through the foreground segmentation algorithm. For example, the third module is any module that applies the foreground segmentation algorithm.

[0098] It should be noted that the terminal extracts multiple different types of features through different modules, obtains richer information, and improves the diversity of feature extraction. In the subsequent process, the fusion of multiple features can provide a more comprehensive image description, which is conducive to the video generation model to learn the original image more comprehensively. At the same time, the modular design makes the image feature extraction process more flexible, facilitates subsequent model adjustment and expansion, and improves the flexibility and adaptability of the model. It should be noted that the first module, the second module, the third module and other modules can be regarded as separate models that do not affect each other and will not be described in detail here.

[0099] In step S305, the terminal copies the latent features, masked features and noise data respectively to obtain a target number of latent features, a target number of masked features and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated.

[0100] In an embodiment of the present disclosure, when the target number has been determined, the target number of images is obtained by predicting the video image of each frame, and a dynamic video is obtained by splicing them together. The video image of each frame is generated based on multiple information, and the multiple information includes implicit features and masked features. Therefore, before the terminal obtains the video image through the video generation model, the terminal copies the implicit features, masked features, etc. that have been obtained, and obtains the target number of implicit features and the target number of masked features. The target number of implicit features and the target number of masked features correspond one to one to a preset number of video frames. In the above manner, the terminal ensures that the multiple information corresponding to the video frame can be obtained in a timely manner when generating the video image of each frame. See Figure 4 As shown, the terminal copies the single implicit feature 403 into the target number of implicit features 405 , and the terminal copies the single masked feature 404 into the target number of masked features 406 .

[0101] In step S306, the terminal performs feature fusion on the target number of implicit features, the target number of masked features, and the target number of noise data, respectively, on the implicit features corresponding to each video frame, the masked features corresponding to the video frame, and the noise data corresponding to the video frame to obtain the intermediate features of the target number.

[0102] In the embodiment of the present disclosure, since latent features and masked features are direct input data of the video generation model, for the data corresponding to each frame of video image, the terminal will splice the latent features, masked features and noise data in the channel dimension, thereby performing feature fusion to obtain intermediate features. Figure 4As shown, the terminal concatenates the target number of implicit features 406 , the target number of masked features 406 , and the target number of noise data 407 in the channel dimension to obtain the target number of intermediate features.

[0103] It should be noted that the terminal copies the implicit features, masked features and noise data to the target number so that each frame of video corresponds to an independent feature representation. This not only increases the controllability of the video generation process, but also makes the video generation of each frame based on more specific and rich feature information, which is conducive to generating more coherent dynamic video, thereby improving the quality and usability of the generated dynamic video.

[0104] In step S307, the terminal uses the intermediate features as input data of the input layer of the first video generation model, uses the camera features and semantic features as data for feature fusion in each layer of the first video generation model, and performs encoding and decoding based on the first video generation model to obtain a dynamic video of the target object. The dynamic video is used to display the target object based on the camera features.

[0105] In the disclosed embodiments, the first video generation model refers to a trained neural network model used in the deduction process. This model is used to predict dynamic video based on the input image. The terminal uses intermediate features as direct input data for the model, and camera features and semantic features as conditional data for the model. Conditional data refers to information provided to the model in addition to direct input data.

[0106] See also Figure 4 As shown in the figure, the model consists of an encoder on the left, which gradually reduces the size of feature maps through multiple convolutional and pooling layers. The decoder on the right gradually increases the size of feature maps through upsampling and convolutional layers. The model also includes a timing module to handle the temporal dependencies between different video frames. This model uses this structure to generate dynamic videos containing a target number of video frames.

[0107] Optionally, the conditional data interacts with each layer of the video generation model by embedding to guide the data processing process and improve model performance and data processing efficiency. For camera features, the terminal injects the camera features into the latent features by point addition before the timing module. The latent features are used to indicate the feature maps extracted by each layer in the model. In other words, the camera features and latent features are fused by adding the corresponding elements one by one, which facilitates the subsequent combination of camera movement and time information in the timing module to generate dynamic videos that are more in line with the display logic. See Figure 4 As shown, semantic features 402 and camera features 408 are used as conditional data to interact with each layer in the video generation model. The model outputs dynamic video 409.

[0108] It should be noted that the terminal generates intermediate features based on implicit features and masked features in combination with noise data, which can more accurately describe the outline and details of the target object, making the subsequently generated dynamic video more accurate and rich in details. At the same time, the terminal uses camera features and semantic features as data for feature fusion in the process of model-generated video, so that the model can adjust the video generation process and data according to the above features, making the video generation process more flexible. In other words, the terminal introduces the above-mentioned multiple features into the video generation model and guides the video generation model to perform denoising, so that the model can predict a dynamic video with high consistency of the target object, rich and accurate details, and reasonable motion trajectory and motion amplitude. The above method improves the efficiency of video generation and is conducive to generating more accurate and higher-quality dynamic videos.

[0109] In some embodiments, the training steps of the first feature extraction model and the first video generation model include the following steps 1 and 2.

[0110] Step 1: Obtain multiple training data sets, which include a first data set, a second data set, and a third data set. The first data set includes multiple sample videos, the second data set includes sample videos of sample objects in a target field, where the target field is the field to which the target object belongs, and the third data set includes sample videos in the second data set that meet preset conditions.

[0111] For ease of description, the first dataset can also be referred to as a general dataset. The sample objects in the first dataset are common objects in life and are divided into multiple fields. For example, the sample objects in the first dataset include butterflies, shirts, lawns, water cups, etc. Optionally, the target field is the e-commerce field. The sample objects in the second dataset are all objects in the e-commerce field. For example, the sample objects in the second dataset include commodities such as shirts and water cups, but do not include natural objects such as butterflies and lawns in nature. The third dataset includes the high-quality data in the second dataset. The sample videos in the third dataset have the characteristics of high definition, high aesthetics, and advanced camera movements.

[0112] Step 2: Based on the first data set, the second data set and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model.

[0113] In some embodiments, the terminal performs multi-stage training on the model by selecting different data sets, as shown in (1)-(3) below.

[0114] (1) The terminal performs a first-stage training on the second feature extraction model and the second video generation model based on the first data set and the second data set to obtain a first-stage feature extraction model and a first-stage video generation model. The dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set.

[0115] The second feature extraction model and the second video generation model are both trained models. The terminal is trained using the first and second datasets, enabling the model to learn common motion patterns of sample objects and camera motion patterns, thereby enhancing the model's generalization capabilities.

[0116] (2) The terminal performs a second-stage training on the feature extraction model of the first stage and the video generation model of the first stage based on the second data set to obtain a second-stage feature extraction model and a second-stage video generation model. The second-stage video generation model is used to generate dynamic videos corresponding to the target domain.

[0117] The terminal fine-tunes the model using the second data set, enabling the model to learn the motion patterns of sample objects in the target area, thereby generating dynamic videos that are more in line with the characteristics of the target area, thereby improving the quality and relevance of the generated videos.

[0118] (3) The terminal performs a third-stage training on the feature extraction model of the second stage and the video generation model of the second stage based on the third data set to obtain a first feature extraction model and a first video generation model. The video score of the dynamic video output by the video generation model of the third stage is higher than the video score of the dynamic video output by the video generation model of the second stage.

[0119] The terminal further fine-tunes the model through the third data set, allowing the model to learn the camera movement rules of a better camera, which is conducive to further optimizing the effect and quality of the generated video.

[0120] It's important to note that through phases 1, 2, and 3 of training, the terminal's model gradually deepens its understanding of information from different data sets, transitioning from generalized learning to in-depth learning in specific areas. This is followed by model fine-tuning and optimization, improving overall training effectiveness and quality. Furthermore, since each phase optimizes for specific training objectives, the model better understands and adapts to the training objectives of each phase, achieving the desired training results more quickly and improving model training efficiency.

[0121] In some embodiments, for any training phase, the terminal updates the model parameters based on the predicted video and the sample video. Accordingly, the terminal inputs the sample video in the training data set into the current video processing model for feature extraction to obtain sample camera features, which are used to represent the trajectory and viewing angle of the camera of the sample video; the terminal inputs the first frame of the sample video into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features, and sample masking features, which are used to represent the potential information of the first frame of the video image, and the sample masking features are used to distinguish the sample object in the first frame of the video image and the part other than the sample object in the first frame of the video image; the terminal inputs the sample semantic features, sample latent features, sample masking features, and sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, which is used to display the sample object based on the sample camera features; based on the error between the predicted video and the sample video, the terminal updates the parameters of the current feature extraction model, the current video generation model, and the current video processing model through the objective function to obtain the feature extraction model of the training phase, the video generation model of the training phase, and the video processing model of the training phase.

[0122] It should be noted that in the video generation method of the embodiment of the present disclosure, in addition to inputting camera features into the video generation model for training, sample videos can also be directly input into the video generation model for training, which will not be elaborated here.

[0123] Among them, the video processing model is used to extract camera features. Optionally, during the training process, the terminal uses a feature extraction tool to extract information from the sample video, which contains at least a 1x3 translation vector and a 3x3 rotation matrix. After flattening the information, the feature extraction network model is used to extract camera features. Among them, the translation vector is used to represent the linear movement distance of the camera along the X, Y, and Z axes in three-dimensional space. The rotation matrix is ​​used to describe the rotation direction and rotation angle of the camera in three-dimensional space. The rotation matrix is ​​also used to convert a point in the camera coordinate system to another coordinate system, which will not be repeated here. Therefore, the camera feature describes how the camera moves in three-dimensional space when shooting video, that is, the way the camera moves.

[0124] It should be noted that, during the video generation process, the subsequent video frames I1, I2, ..., I N, thereby obtaining a dynamic video. Among them, the initial reference frame of the target object is the original image, which is also the first frame of video image in the dynamic video. The video generation method in the embodiment of the present disclosure adopts a diffusion model, and the objective function of the model is used to indicate the difference between the predicted noise of the model and the actual noise. Among them, the predicted noise of the model is determined by the time step, implicit features, initial reference frame and camera features. The time step is used to represent the process of the diffusion process from the initial state to the noisy state and from the noisy state to the clear state. The expected values ​​of the implicit features, initial reference frame, camera features, time step and noise data are taken to obtain the total loss value corresponding to the model. That is, compared with the objective function of the traditional diffusion model, the objective function of the model in the embodiment of the present disclosure also includes camera features.

[0125] It should be noted that during any training phase, the terminal simultaneously extracts the camera features of the sample video and the semantic, implicit, and masked features of the first frame of the video image, enabling the model to obtain richer and more comprehensive information, which is conducive to maintaining consistency in the video generation process. At the same time, by introducing camera features during the training process, the terminal enables the model to simulate the movement and behavior of the camera in the display scenario of the target object, improving the quality and usability of the generated video. In addition, the terminal evaluates model performance by the error between the predicted video and the sample video, and continuously adjusts and optimizes the model parameters to improve model performance.

[0126] The disclosed embodiment provides a video generation method, in which multiple image features and camera features, such as semantic features, latent features, and masked features, extracted from an original image are used as input to a first video generation model, so that the first video generation model can not only display the target object in the original image in a dynamic video, but also learn the camera features, thereby making the camera trajectory and camera perspective of the dynamic video more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this method introduces multiple image features and camera features, so that the first video generation model uses the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, thereby improving the video quality and availability.

[0127] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0128] Figure 5 FIG. 1 is a block diagram of a video generating apparatus according to an exemplary embodiment. Figure 5 As shown, the device includes: a determining unit 501, an extracting unit 502 and a generating unit 503.

[0129] A determining unit 501 is configured to obtain camera features associated with an original image, where the original image includes a target object, and the camera features are used to represent a trajectory and a viewing angle of the camera;

[0130] Extraction unit 502 is configured to input the original image into a first feature extraction model to perform feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish a target object in the original image from parts of the original image other than the target object;

[0131] The generation unit 503 is configured to input the semantic features, implicit features, masked features and camera features into the first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

[0132] In some embodiments, the determination unit 501 is configured to determine the camera feature associated with the item type as the camera feature associated with the original image based on the item type corresponding to the original image, where the item type is the type of the target object contained in the original image; or, in the case where multiple preset camera features have been set, in response to a selection operation on any preset camera feature, determine the preset camera feature as the camera feature associated with the original image, where different preset camera features are used to represent different camera trajectories and viewing angles.

[0133] In some embodiments, Figure 6 is a block diagram of another video generating device according to an exemplary embodiment. Figure 6 As shown, the generating unit 503 includes:

[0134] The fusion subunit 5031 is configured to perform feature fusion on the latent features, the masked features, and the noise data to obtain intermediate features;

[0135] The generation subunit 5032 is configured to use the intermediate features as input data of the input layer of the first video generation model, use the camera features and semantic features as data for feature fusion in each layer of the first video generation model, encode and decode based on the first video generation model, and obtain a dynamic video of the target object.

[0136] In some embodiments, the fusion sub-unit 5031 is configured to copy the latent features, masked features and noise data respectively to obtain a target number of latent features, a target number of masked features and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated; for the target number of latent features, the target number of masked features and the target number of noise data, the target number of intermediate features are obtained for the latent features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame.

[0137] In some embodiments, the apparatus further comprises:

[0138] An acquisition unit 601 is configured to acquire multiple training data sets, the multiple training data sets including a first data set, a second data set, and a third data set, the first data set including multiple sample videos, the second data set including sample videos of sample objects in a target domain, where the target domain is the domain to which the target object belongs, and the third data set including sample videos in the second data set that meet preset conditions;

[0139] The training unit 602 is configured to train the second feature extraction model and the second video generation model based on the first data set, the second data set and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

[0140] In some embodiments, the training unit 602 is configured to perform a first-stage training on the second feature extraction model and the second video generation model based on the first data set and the second data set to obtain a first-stage feature extraction model and a first-stage video generation model, and the dynamic video output by the first-stage video generation model is used to reflect the camera features indicated by the first data set and the camera features indicated by the second data set; based on the second data set, perform a second-stage training on the first-stage feature extraction model and the first-stage video generation model to obtain a second-stage feature extraction model and a second-stage video generation model, and the second-stage video generation model is used to generate dynamic video corresponding to the target field; based on the third data set, perform a third-stage training on the second-stage feature extraction model and the second-stage video generation model to obtain a first feature extraction model and a first video generation model, and the video score of the dynamic video output by the third-stage video generation model is higher than the video score of the dynamic video output by the second-stage video generation model.

[0141] In some embodiments, the training unit 602 is configured to, for any training stage, input the sample video in the training data set into the current video processing model for feature extraction to obtain sample camera features, which are used to represent the trajectory and viewing angle of the camera of the sample video; input the first frame video image of the sample video into the current feature extraction model for feature extraction to obtain sample semantic features, sample latent features and sample masking features, which are used to represent the potential information of the first frame video image, and the sample masking features are used to distinguish the sample object in the first frame video image and the part of the first frame video image other than the sample object; input the sample semantic features, sample latent features, sample masking features and sample camera features into the current video generation model for decoding and encoding to obtain a predicted video, which is used to display the sample object based on the sample camera features; based on the error between the predicted video and the sample video, update the parameters of the current feature extraction model, the current video generation model and the current video processing model through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

[0142] The disclosed embodiment provides a video generation device, in which the terminal uses multiple image features and camera features extracted from the original image, such as semantic features, latent features, and masked features, as input to a first video generation model, so that the first video generation model can not only display the target object in the original image in the dynamic video, but also learn the camera features, so that the camera trajectory and camera perspective of the dynamic video are more consistent with the display logic. Compared with the traditional method of directly inputting semantic features into the first video generation model, this device introduces multiple image features and camera features, so that the first video generation model takes the target object as the main body, generates a dynamic video with richer details and more consistent with the display logic, thereby improving the video quality and availability.

[0143] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0144] It should be noted that the video generation device provided in the above embodiment is merely an example of the division of the above functional units. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the video generation device provided in the above embodiment and the video generation method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0145] Regarding the video generating device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0146] Figure 7 FIG. 7 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 700 includes a processor 701 and a memory 702 .

[0147] The processor 701 may include one or more processing cores, such as a 4-core processor, a 7-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0148] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is executed by the processor 701 to implement the video generation method provided in the method embodiment of the present disclosure.

[0149] In some embodiments, electronic device 700 may optionally include a peripheral device interface 703 and at least one peripheral device. Processor 701, memory 702, and peripheral device interface 703 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 703 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, and a power supply 708.

[0150] The peripheral device interface 703 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0151] The RF circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 704 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 704 can communicate with other electronic devices via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in this disclosure.

[0152] Display screen 705 is used to display a user interface (UI). This UI can include graphics, text, icons, videos, or any combination thereof. When display screen 705 is a touchscreen display, it can also capture touch signals on or above the surface of display screen 705. These touch signals can be input as control signals to processor 701 for processing. Display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be a single display screen 705, located on the front panel of electronic device 700. In other embodiments, there can be at least two display screens 705, located on different surfaces of electronic device 700 or in a foldable design. In still other embodiments, display screen 705 can be a flexible display, located on a curved or foldable surface of electronic device 700. Display screen 705 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0153] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0154] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 701 for processing, or to be input into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the electronic device 700. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0155] Power supply 708 is used to power the various components of electronic device 700. Power supply 708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0156] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the electronic device 700, and the electronic device 700 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0157] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 including instructions. The instructions can be executed by a processor 701 of an electronic device 700 to perform the above-described video generation method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] A computer program product includes a computer program, which implements the above-mentioned video generation method when executed by a processor.

[0159] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0160] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that: The method comprises: Obtaining camera features associated with an original image, wherein the original image includes a target object, the camera features being used to represent a trajectory and a viewing angle of the camera; Inputting the original image into a first feature extraction model for feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from parts of the original image other than the target object; The semantic features, the latent features, the masked features, and the camera features are input into a first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

2. The video generation method according to claim 1, wherein: The acquiring of camera features associated with the original image includes: Based on the item type corresponding to the original image, determining the camera feature associated with the item type as the camera feature associated with the original image, the item type being the type of the target object contained in the original image; or In the case where multiple preset camera features are set, in response to a selection operation on any preset camera feature, the preset camera feature is determined as the camera feature associated with the original image, and different preset camera features are used to represent different camera trajectories and viewing angles.

3. The video generation method according to claim 1, wherein: The step of inputting the semantic features, the implicit features, the masked features, and the camera features into a first video generation model for decoding and encoding includes: Performing feature fusion on the latent feature, the masked feature and the noise data to obtain an intermediate feature; The intermediate features are used as input data of the input layer of the first video generation model, the camera features and the semantic features are used as data for feature fusion in each layer of the first video generation model, and encoding and decoding are performed based on the first video generation model to obtain a dynamic video of the target object.

4. The video generation method according to claim 3, wherein: The step of fusing the latent features, the masked features, and the noise data to obtain intermediate features includes: The latent features, the masked features, and the noise data are respectively copied to obtain a target number of latent features, a target number of masked features, and a target number of noise data, where the target number is equal to a preset number of video frames of the video to be generated; For the implicit features of the target number, the masked features of the target number and the noise data of the target number, feature fusion is performed on the implicit features corresponding to each video frame, the masked features corresponding to the video frame and the noise data corresponding to the video frame to obtain the intermediate features of the target number.

5. The video generation method according to claim 1, wherein: The training steps of the first feature extraction model and the first video generation model include: Acquire multiple training data sets, the multiple training data sets including a first data set, a second data set, and a third data set, the first data set including multiple sample videos, the second data set including sample videos of sample objects in a target domain, the target domain being the domain to which the target objects belong, and the third data set including sample videos in the second data set that meet preset conditions; Based on the first data set, the second data set and the third data set, the second feature extraction model and the second video generation model are trained through multiple training stages to obtain the first feature extraction model and the first video generation model, and different training stages correspond to different training data sets.

6. The video generation method according to claim 5, characterized in that The method of training a second feature extraction model and a second video generation model based on the first data set, the second data set, and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model includes: Based on the first data set and the second data set, performing a first-stage training on the second feature extraction model and the second video generation model to obtain a first-stage feature extraction model and a first-stage video generation model, wherein the first-stage video generation model outputs a dynamic video that reflects the camera features indicated by the first data set and the camera features indicated by the second data set; Based on the second data set, performing a second-stage training on the feature extraction model of the first stage and the video generation model of the first stage to obtain a second-stage feature extraction model and a second-stage video generation model, wherein the second-stage video generation model is used to generate a dynamic video corresponding to the target domain; Based on the third data set, the feature extraction model of the second stage and the video generation model of the second stage are trained for the third stage to obtain the first feature extraction model and the first video generation model. The video score of the dynamic video output by the video generation model of the third stage is higher than the video score of the dynamic video output by the video generation model of the second stage.

7. The video generation method according to claim 5, characterized in that: The method of training a second feature extraction model and a second video generation model based on the first data set, the second data set, and the third data set through multiple training stages to obtain the first feature extraction model and the first video generation model includes: For any training stage, the sample video in the training data set is input into the current video processing model for feature extraction to obtain sample camera features, where the sample camera features are used to represent the trajectory and viewing angle of the camera of the sample video; Inputting a first frame of the sample video into a current feature extraction model for feature extraction to obtain sample semantic features, sample latent features, and sample masking features, wherein the sample latent features are used to represent potential information of the first frame of the video image, and the sample masking features are used to distinguish a sample object in the first frame of the video image from a portion of the first frame of the video image other than the sample object; Inputting the sample semantic features, the sample latent features, the sample masked features, and the sample camera features into a current video generation model for decoding and encoding to obtain a predicted video, wherein the predicted video is used to display the sample object based on the sample camera features; Based on the error between the predicted video and the sample video, the parameters of the current feature extraction model, the current video generation model and the current video processing model are updated through the objective function to obtain the feature extraction model of the training stage, the video generation model of the training stage and the video processing model of the training stage.

8. A video generating device, characterized in that: The device comprises: a determining unit configured to obtain camera features associated with an original image, wherein the original image includes a target object, the camera features being used to represent a trajectory and a viewing angle of the camera; an extraction unit configured to input the original image into a first feature extraction model to perform feature extraction to obtain semantic features, latent features, and masked features, wherein the latent features are used to represent potential information of the original image, and the masked features are used to distinguish the target object in the original image from a portion of the original image other than the target object; The generation unit is configured to input the semantic features, the implicit features, the masked features and the camera features into a first video generation model for decoding and encoding to obtain a dynamic video, which is used to display the target object based on the camera features.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video generating method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the video generation method according to any one of claims 1 to 7 is implemented.

Images