Video generation method and device based on physical attribute enhancement, equipment and medium

Abstract

The invention discloses a video generation method and device based on physical attribute enhancement, equipment and a medium, and the method comprises the steps: obtaining an image feature and a summary description feature corresponding to an initial frame image, and the summary description feature is used for representing the physical motion of an object in the initial frame image; obtaining a target semantic enhanced physical attribute feature corresponding to the initial frame image; inputting the image features, the summary description features and the target semantic enhancement physical attribute features to a video diffusion model to obtain a sequence image frame; and inputting the sequence image frame to a video decoder to obtain a target video. According to the method, the physical motion of the object in the initial frame image is predicted according to the image feature, the summary description feature and the target semantic enhancement physical attribute feature corresponding to the initial frame image based on the video diffusion model, and the corresponding target video is generated, so that the matching degree of the generated video and the user demand is improved.

Description

Technical Field

[0001] This application belongs to the field of video generation technology, and in particular relates to a video generation method, apparatus, device and medium based on physical property enhancement. Background Technology

[0002] In recent years, video-generated world models have been widely used in fields such as virtual reality, autonomous driving, and robot planning.

[0003] However, existing world models generally suffer from a lack of fine-grained perception of physical properties, making it difficult to accurately simulate the dynamic behavior of objects in the real world, such as the relationship between the sliding, deformation, and stopping position of a soft object on a slope, resulting in generated videos that do not match user needs. Summary of the Invention

[0004] In view of the above, embodiments of this application provide a video generation method, apparatus, device and medium based on physical property enhancement to overcome the problems of the prior art.

[0005] In a first aspect, embodiments of this application provide a video generation method based on physical attribute enhancement, including: Obtain image features and summary description features corresponding to the initial frame image, wherein the summary description features are used to characterize the physical motion of objects in the initial frame image; Obtain the target semantic enhanced physical attribute features corresponding to the initial frame image. The target semantic enhanced physical attribute features are used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The target semantic enhanced physical attribute features have the same feature space as the summary description features. The image features, the summary description features, and the target semantic enhancement physical attribute features are input into the video diffusion model to obtain a sequence of image frames. The sequence of image frames is used to characterize the physical motion of the object, and the first frame of the sequence of image frames is the initial frame image. The sequence of image frames is input to the video decoder to obtain the target video.

[0006] In some optional embodiments, obtaining the target semantic enhancement physical attribute features corresponding to the initial frame image includes: Obtain the target physical property characteristics; Obtain the target physical semantic features; Based on the perceiver-resampler, the target physical attribute features are mapped to the physical semantic space of the target physical semantic features to obtain the initial semantically enhanced physical attribute features; The initial semantically enhanced physical attribute features are mapped to the summary feature space of the summary description features based on the multilayer perceptron to obtain the target semantically enhanced physical attribute features.

[0007] In some optional embodiments, obtaining the target physical attribute features includes: Obtain a physical attribute text description, which is used to characterize the physical attributes of the initial frame image; Input the physical attribute text description into the text encoder to obtain the target physical attribute features.

[0008] In some optional embodiments, obtaining the target physical semantic features includes: Obtain a physical semantic query embedding set, which is used to characterize the physical application domain of the initial frame image; Input the physical semantic query embedding set into the text encoder to obtain the initial physical semantic features; The initial physical semantic features are query-embedded to generate query-embedded physical semantic features; Principal component analysis is performed on the embedded physical semantic features of the query to obtain the target physical semantic features.

[0009] In some optional embodiments, the step of query embedding the initial physical semantic features to generate query embedded physical semantic features includes: The first dimension of the initial physical semantic features is averaged to obtain the query embedded physical semantic features.

[0010] In some optional embodiments, obtaining the physical semantic query embedding set includes: Obtain textual descriptions of physical application domains; The physical application domain text description is input into the physical semantic query embedding set generation model to obtain the physical semantic query embedding set. The physical semantic query embedding set generation model is obtained by training a large language model based on a sample set labeled with physical application domain text description tags and physical semantic query embedding set tags.

[0011] In some optional embodiments, the resampling perceiver-resampler includes a linear projection layer, a cross-modal attention layer, and a normalization layer connected in sequence.

[0012] Secondly, embodiments of this application provide a video generation apparatus based on physical property enhancement, comprising: The first acquisition module is used to acquire image features and summary description features corresponding to the initial frame image, wherein the summary description features are used to characterize the physical motion of objects in the initial frame image; The second acquisition module is used to acquire the target semantic enhanced physical attribute features corresponding to the initial frame image. The target semantic enhanced physical attribute features are used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The target semantic enhanced physical attribute features have the same feature space as the summary description features. The first input module is used to input the image features, the summary description features and the target semantic enhancement physical attribute features into the video diffusion model to obtain a sequence of image frames. The sequence of image frames is used to characterize the physical motion of the object. The first frame of the sequence of image frames is the initial frame image. The second input module is used to input the sequence of image frames to the video decoder to obtain the target video.

[0013] Thirdly, embodiments of this application provide an electronic device, including: Memory; One or more processors are coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the physically enhanced video generation method as provided in the first aspect above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the video generation method based on physical attribute enhancement provided in the first aspect above.

[0015] Fifthly, embodiments of this application provide a computer program product that, when run on a computer device, causes the computer device to execute the video generation method based on physical attribute enhancement as provided in the first aspect above.

[0016] The solution provided in this application obtains image features and summary description features corresponding to the initial frame image. The summary description features are used to characterize the physical motion of objects in the initial frame image. It also obtains target semantic-enhanced physical attribute features corresponding to the initial frame image. These target semantic-enhanced physical attribute features are used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The feature space of the target semantic-enhanced physical attribute features is the same as that of the summary description features. These features are then input into a video diffusion model to obtain a sequence of image frames. The sequence of image frames is used to characterize the physical motion of objects. The first frame of the sequence of image frames is the initial frame image. Finally, the sequence of image frames is input into a video decoder to obtain the target video. This solution achieves prediction of the physical motion of objects in the initial frame image based on the image features, summary description features, and target semantic-enhanced physical attribute features corresponding to the initial frame image, and generates the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This illustration shows a flowchart of a video generation method based on physical property enhancement provided in an embodiment of this application.

[0019] Figure 2 This illustration shows a flowchart of the video generation process in the video generation method based on physical attribute enhancement provided in an embodiment of this application.

[0020] Figure 3 This illustration shows another flowchart of the video generation method based on physical property enhancement provided in an embodiment of this application.

[0021] Figure 4 A structural block diagram of a video generation apparatus based on physical property enhancement provided in an embodiment of this application is shown.

[0022] Figure 5 A functional block diagram of an electronic device provided in an embodiment of this application is shown.

[0023] Figure 6This application illustrates a computer-readable storage medium for storing or carrying program code that implements the physically enhanced video generation method according to an embodiment of this application.

[0024] Figure 7 This application illustrates a computer program product for storing or carrying program code that implements the physically enhanced video generation method according to the embodiments of this application. Detailed Implementation

[0025] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0027] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] In recent years, video-generated world models have been widely used in fields such as virtual reality, autonomous driving, and robot planning.

[0031] However, existing world models generally suffer from a lack of fine-grained perception of physical properties, making it difficult to accurately simulate the dynamic behavior of objects in the real world, such as the relationship between the sliding, deformation, and stopping position of a soft object on a slope, resulting in generated videos that do not match user needs.

[0032] To address the aforementioned issues, the present application provides a video generation method, apparatus, device, and medium based on enhanced physical attributes. This method acquires image features and summary description features corresponding to an initial frame image. The summary description features characterize the physical motion of objects in the initial frame image. It also acquires target semantic-enhanced physical attribute features corresponding to the initial frame image. These target semantic-enhanced physical attribute features characterize the target physical attribute features and target physical semantic features of the initial frame image. The target semantic-enhanced physical attribute features share the same feature space as the summary description features. The image features, summary description features, and target semantic-enhanced physical attribute features are input into a video diffusion model to obtain a sequence of image frames. The sequence of image frames characterizes the physical motion of objects, with the first frame being the initial frame image. The sequence of image frames is then input into a video decoder to obtain a target video. This method, based on a video diffusion model, predicts the physical motion of objects in an initial frame image using the image features, summary description features, and target semantic-enhanced physical attribute features, and generates the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs.

[0033] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] The video generation method provided in this application embodiment can be applied to a processing device, which can be any of the following: a server or a terminal device. No limitation is made here, and the specific device can be set according to actual needs.

[0035] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), big data, and artificial intelligence platforms.

[0036] Terminal devices can be mobile terminal devices (e.g., in-vehicle terminals, PDAs, tablet PCs, laptops, etc.) or fixed terminal devices (desktop computers, smart panels, etc.).

[0037] Please see Figure 1 This document illustrates a flowchart of a video generation method based on physical property enhancement according to an embodiment of this application. In a specific embodiment, the video generation method based on physical property enhancement can be applied to a processing device. The following will use a processing device as an example to illustrate this method. Figure 1 The process shown is described in detail. The video generation method based on physical property enhancement may include the following steps 101 to 104.

[0038] Step 101: Obtain the image features and summary description features corresponding to the initial frame image.

[0039] In this embodiment of the application, when a user needs to generate a video, he / she can send a generation instruction to the processing device. After receiving the generation instruction, the processing device can obtain the image features and summary description features corresponding to the initial frame image.

[0040] The initial frame image can be any image containing an object. The object in the initial frame image can be at least one of the following: a person, an animal, or other objects. The summary description features can be used to characterize the physical motion of the object in the initial frame image.

[0041] Specifically, when a user needs to generate a video, they can send a generation command to the processing device. After receiving the generation command, the processing device can obtain the initial frame image and its corresponding summary text description, and obtain image features based on the initial frame image and summary description features based on the summary text description.

[0042] The summary text description can be used to describe the physical changes of objects in the initial frame image; for example, the summary text description can be a thrown object.

[0043] Regarding the process by which the aforementioned processing device acquires the initial frame image and its corresponding summary text description, in some embodiments, the processing device may generate a prompt message and receive the initial frame image and its corresponding summary text description uploaded by the user based on the prompt message.

[0044] The prompt information can be used to prompt the user to upload the initial frame image and its corresponding summary text description to the processing device. The prompt information can be at least one of text prompt information, sound prompt information, or light prompt information, etc., without limitation here.

[0045] Regarding the process by which the aforementioned processing device acquires the initial frame image and its corresponding summary text description, in some embodiments, the processing device may pre-store the initial frame image and its corresponding summary text description, and the processing device may read the pre-stored initial frame image and its corresponding summary text description.

[0046] Regarding the process by which the above-mentioned processing device obtains image features based on the initial frame image, in some embodiments, the processing device can input the initial frame image to the image encoder. After receiving the initial frame image, the image encoder can encode the initial frame image to obtain image features.

[0047] Regarding the process by which the above-mentioned processing device obtains summary description features based on the summary text description, in some embodiments, the processing device may input the summary text description to the text encoder. After receiving the summary text description, the text encoder may encode the summary text description to obtain the summary description features.

[0048] The text encoder can be a pre-trained language model (e.g., the T5 model).

[0049] Step 102: Obtain the target semantic enhancement physical attribute features corresponding to the initial frame image.

[0050] In this embodiment of the application, the target semantic enhanced physical attribute features can be used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The target semantic enhanced physical attribute features have the same feature space as the summary description features.

[0051] The processing device can acquire target physical attribute features and target physical semantic features. Based on a resampler-resampler, it maps the target physical attribute features to the physical semantic space of the target physical semantic features, obtaining initial semantically enhanced physical attribute features. Then, based on a multilayer perceptron, it maps the initial semantically enhanced physical attribute features to the summary feature space of the summary description features, obtaining target semantically enhanced physical attribute features. By fusing the target physical attribute features and target physical semantic features using the resampler-resampler and the multilayer perceptron, and aligning the physical attributes with the physical semantics, the device can control the motion trajectory of objects in the generated video, thereby improving the matching degree between the generated video and user needs.

[0052] The target physical attribute features can be used to characterize the physical properties of the initial frame image. The resampling perceptron (perceiver-resampler) can include a linear projection layer, a cross-modal attention layer, and a normalization layer connected in sequence.

[0053] Linear projection layers can be used to linearly project target physical attribute features onto target physical semantic features to obtain projected features. Cross-modal attention layers can be used to perform cross-modal computation on the projected features to obtain cross-modal features. Normalization layers can be used to normalize the cross-modal features to obtain initial semantically enhanced physical attribute features.

[0054] Regarding the process by which the aforementioned processing device acquires the target physical attribute features, in some embodiments, the processing device can acquire a physical attribute text description and input the physical attribute text description into a text encoder. After receiving the physical attribute text description, the text encoder can encode the physical attribute text description to obtain the target physical attribute features. Encoding the physical attribute text description using a text encoder to obtain the target physical attribute features improves the accuracy of the target physical attribute features.

[0055] The physical attribute text description is used to characterize the physical properties of the initial frame image. For example, the physical attribute text description can be... P ( f , ζ , v Physical property text description P ( f , ζ , v ,…)=“Dynamic friction coefficient f linear stiffness ζ Horizontal speed v ,…”.

[0056] Processing devices can use T5 models to describe physical properties in text. P ( f , ζ , v Encode the target physical attribute features by encoding (…). e ´ p .

[0057] e ´ p =T5( P ( f , ζ , v ,…))∈ R Np×512 , R For the feature space, N p ×512 is the dimension. N p This represents the number of physical attribute text tokens, or the number of terms.

[0058] Regarding the process of the aforementioned processing device acquiring target physical semantic features, in some embodiments, the processing device can acquire a physical semantic query embedding set and input it into a text encoder. After receiving the physical semantic query embedding set, the text encoder can encode the set to obtain initial physical semantic features, perform query embedding on these initial features to generate query-embedded physical semantic features, and perform principal component analysis on these features to obtain the target physical semantic features. Constructing target physical semantic features based on the physical semantic query embedding set improves the accuracy of the target physical semantic features.

[0059] Principal Components Analysis (PCA) can select the lowest number of features with the lowest similarity among the query's embedded physical semantic features based on the feature similarity of the query's embedded physical semantic features, and compress the selected features to obtain the target physical semantic features.

[0060] A physics semantic query embedding set can be a set of descriptions of basic physical concepts. As an example, a physics semantic query embedding set can be... δ Physical semantic query embedding set δ It can include n A description of a fundamental physical concept, namely δ = S i , i =1, 2, ..., n .

[0061] For example, physical semantic query embedding set S i It can cover fundamental mechanical quantities and be used to guide semantic reconstruction.

[0062] .

[0063] Processing devices can use the T5 model to embed physical semantic queries into a set. S i Encode the initial physical semantic features T5. S i ).

[0064] The processing device can average the first dimension of the initial physical semantic features to obtain the query-embedded physical semantic features. Averaging the first dimension of the initial physical semantic features reduces the dimensionality of the query-embedded physical semantic features, thereby reducing the computational load during video generation and improving the efficiency of video generation.

[0065] As an example, the processing device can process the initial physical semantic features T5 (S i The first dimension of the feature is averaged to obtain the query embedded physical semantic feature Ω.

[0066] For example, the processing device can process the initial physical semantic features T5 ( S i The average feature is obtained by averaging the first dimension of each feature in the matrix. ω i =mean0(T5( S i ))∈ R 512 mean0() is an operation that calculates the average of the first dimension of the feature.

[0067] And it can be based on each average feature ω i Construct query embedding physical semantic features Ω, Ω={ ω i} Ns i=1 ∈ R 512 , N s The number of physical concepts.

[0068] The processing device can perform principal component analysis on the query embedded physical semantic features Ω, compressing the query embedded physical semantic features Ω into... R 16×512 , thus obtaining the target physical semantic features Ω´.

[0069] The processing device can use a linear projection layer to represent the physical properties of the target. e ´ p Linear projection is performed onto the target physical semantic feature Ω´ to obtain the projected feature Lin( e ´ p ), and based on the cross-modal attention layer, the projected features Lin ( e ´ p Cross-modal computation is performed to obtain the cross-modal features Atten(Ω´, Lin( e ´ p )), and based on the normalization layer, the cross-modal features Atten(Ω´,Lin( e ´ p Normalization is performed to obtain the initial semantically enhanced physical attribute features. e ´´ p .

[0070] e ´´ p =LN(Atten(Ω´,Lin( e´ p ))).

[0071] Lin() is a linear projection layer that can be used to... e ´ p The mapping is to key-value pairs; Atten() is a cross-modal attention layer, and LN() is a normalization layer.

[0072] Processing devices can enhance initial semantics with physical attribute features based on multilayer perceptrons. e ´´ p Mapping to the summary feature space of the summary description features yields the target semantically enhanced physical attribute features. e p .

[0073] e p = L proj ( e ´´ p ), L proj () represents the mapping operation of a multilayer perceptron.

[0074] The processing device can acquire a physical application domain text description and input the physical application domain text description into the physical semantic query embedding set generation model to obtain the physical semantic query embedding set.

[0075] Among them, the physical semantic query embedding set generation model can be obtained by training a large language model (LLM) based on a sample set labeled with text description tags of physical application domain and physical semantic query embedding set tags.

[0076] The processing device can receive text descriptions of physical application areas manually entered by the user, or text descriptions of physical application areas entered by the user via voice, etc., without limitation.

[0077] As an example, the text description in the physical application field could be a description of a factory handling robot or a shopping mall service robot, etc., without limitation.

[0078] The tags for the physical semantic query embedding set can be: Mass: The total amount of material in the robot body or payload, affecting stability and energy consumption; Volume: The size of the three-dimensional space occupied by the robot determines its ability to move through factory aisles; Density: The mass per unit volume of robot materials or loads, affecting the overall weight distribution; Length: The dimension of the robot along its main axis, used to evaluate path planning and turning radius; Width: The robot's lateral dimension, which affects its maneuverability in narrow areas; Height: The robot's vertical dimension, which determines its ability to pass through doorways or shelves.

[0079] Step 103: Input image features, summary description features, and target semantic enhancement physical attribute features into the video diffusion model to obtain a sequence of image frames.

[0080] In this embodiment of the application, after the processing device obtains the target semantic enhancement physical attribute features corresponding to the initial frame image, it can input the image features, summary description features and target semantic enhancement physical attribute features into the video diffusion model. After receiving the image features, summary description features and target semantic enhancement physical attribute features, the video diffusion model can output the sequence of image frames.

[0081] Among them, the sequence of image frames can be used to characterize the physical motion of the object. The first frame of the sequence of image frames can be the initial frame image, and the second to the last frame of the sequence of image frames are latent feature images predicted based on the previous frame image.

[0082] The video diffusion model can be a Transformer-based noise diffusion model, while the visual diffusion model can perform reverse diffusion of image features based on summary description features and target semantic enhancement physical attribute features according to the preset number of training steps, to obtain a sequence of image frames.

[0083] Step 104: Input the sequence of image frames to the video decoder to obtain the target video.

[0084] In this embodiment, the processing device inputs image features, summary description features, and target semantic-enhanced physical attribute features into the video diffusion model to obtain a sequence of image frames. The device can then input these image frames into the video decoder. Upon receiving the image frames, the video decoder can decode them to obtain the target video. This achieves the prediction of the physical motion of objects in the initial frame image based on the image features, summary description features, and target semantic-enhanced physical attribute features corresponding to the initial frame image, and generates the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs.

[0085] In one application scenario, such as Figure 2 As shown, the physical attribute text description can be "dynamic friction coefficient", "linear stiffness" and "velocity", and the physical semantic query embedding set can be "inertia: the ability of an object to resist changes in its state of motion", "momentum: the product of mass and velocity" and "acceleration: the change in velocity per unit time".

[0086] The processing device can encode the text description of physical attributes based on the first text encoder to obtain the target physical attribute features.

[0087] The processing device can encode the physical semantic query embedding set based on the second text encoder, which is the initial physical semantic feature.

[0088] The processing device can perform query embedding on the initial physical semantic features to generate query embedded physical semantic features.

[0089] The processing device can perform principal component analysis on the query embedded physical semantic features based on the principal component analysis module to obtain the target physical semantic features.

[0090] The processing device can map the target physical attribute features to the physical semantic space of the target physical semantic features based on the resampler-resampler to obtain the initial semantically enhanced physical attribute features.

[0091] The processing device can map the initial semantically enhanced physical attribute features to the summary feature space of the summary description features based on the multilayer perceptron to obtain the target semantically enhanced physical attribute features.

[0092] The processor can encode the summary text description based on a third text encoder to obtain summary description features.

[0093] The processor can encode the initial frame image based on the image encoder to obtain image features.

[0094] The processor can perform reverse diffusion of image features based on a video diffusion model, according to the target semantic enhancement physical attribute features and summary description features, to obtain a sequence of image frames.

[0095] The processor can decode the sequence of image frames based on the video decoder to obtain the target video.

[0096] The solution provided in this application obtains image features and summary description features corresponding to the initial frame image. The summary description features are used to characterize the physical motion of objects in the initial frame image. It also obtains target semantic-enhanced physical attribute features corresponding to the initial frame image. These target semantic-enhanced physical attribute features are used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The feature space of the target semantic-enhanced physical attribute features is the same as that of the summary description features. These features are then input into a video diffusion model to obtain a sequence of image frames. The sequence of image frames is used to characterize the physical motion of objects. The first frame of the sequence of image frames is the initial frame image. Finally, the sequence of image frames is input into a video decoder to obtain the target video. This solution achieves prediction of the physical motion of objects in the initial frame image based on the image features, summary description features, and target semantic-enhanced physical attribute features corresponding to the initial frame image, and generates the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs.

[0097] Please see Figure 3 This document illustrates a flowchart of a video generation method based on physical property enhancement, provided in another embodiment of this application. In a specific embodiment, the video generation method based on physical property enhancement can be applied to a processing device. The following will use a processing device as an example to illustrate this method. Figure 3 The process shown is described in detail. The video generation method based on physical attribute enhancement may include the following steps 201 to 205.

[0098] Step 201: Obtain the image features and summary description features corresponding to the initial frame image.

[0099] Step 202: Obtain the target semantic enhancement physical attribute features corresponding to the initial frame image.

[0100] Step 203: Input image features, summary description features, and target semantic enhancement physical attribute features into the video diffusion model to obtain a sequence of image frames.

[0101] Step 204: Input the sequence of image frames to the video decoder to obtain the target video.

[0102] In this embodiment, steps 201, 202, 203 and 204 can be referred to the corresponding steps in the previous embodiments, and will not be repeated here.

[0103] Step 205: Send the target video to the display screen.

[0104] In this embodiment, the processing device inputs a sequence of image frames to the video decoder to obtain the target video, and then sends the target video to the display screen. After receiving the target video, the display screen can play the target video so that the user can determine whether the target video meets the user's needs based on the playback content, thereby improving the user experience during the video generation process.

[0105] The display screen can be any of the following: a liquid crystal display (LCD), a light-emitting diode (LED) display screen, or an organic light-emitting diode (OLED) display screen; there is no limitation on this.

[0106] The solution provided in this embodiment obtains the image features and summary description features corresponding to the initial frame image, and obtains the target semantic-enhanced physical attribute features corresponding to the initial frame image. These features are then input into a video diffusion model to obtain a sequence of image frames. The sequence of image frames is then input into a video decoder to obtain the target video, and the target video is sent to the display screen. This achieves the prediction of the physical motion of objects in the initial frame image based on the image features, summary description features, and target semantic-enhanced physical attribute features corresponding to the initial frame image, and the generation of the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs.

[0107] The generated target video is then sent to the display screen so that the screen can play the target video. This allows users to determine whether the target video meets their needs based on the content of the video, thus improving the user experience during the video generation process.

[0108] Please see Figure 4 This document illustrates a physically enhanced video generation apparatus 300 according to an embodiment of this application. In a specific embodiment, the physically enhanced video generation apparatus 300 can be applied to a processing device. The following will use a processing device as an example to illustrate this. Figure 4 The video generation device 300 based on physical property enhancement shown will be described in detail. The video generation device 300 based on physical property enhancement may include a first acquisition module 301, a second acquisition module 302, a first input module 303, and a second input module 304.

[0109] The first acquisition module 301 can be used to acquire image features and summary description features corresponding to the initial frame image. The summary description features can be used to characterize the physical motion of the object in the initial frame image. The second acquisition module 302 can be used to acquire target semantic enhanced physical attribute features corresponding to the initial frame image. The target semantic enhanced physical attribute features can be used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The feature space of the target semantic enhanced physical attribute features and the summary description features can be the same. The first input module 303 can be used to input image features, summary description features, and target semantic enhanced physical attribute features into the video diffusion model to obtain a sequence of image frames. The sequence of image frames can be used to characterize the physical motion of the object. The first image of the sequence of image frames can be the initial frame image. The second input module 304 can be used to input the sequence of image frames into the video decoder to obtain the target video.

[0110] In some implementations, the second acquisition module 302 may include a first acquisition submodule, a second acquisition submodule, a first mapping submodule, and a second mapping submodule.

[0111] The first acquisition submodule can be used to acquire target physical attribute features; the second acquisition submodule can be used to acquire target physical semantic features; the first mapping submodule can be used to map the target physical attribute features to the physical semantic space of the target physical semantic features based on the resampling perceptron perceptron, to obtain initial semantically enhanced physical attribute features; the second mapping submodule can be used to map the initial semantically enhanced physical attribute features to the summary feature space of the summary description features based on the multilayer perceptron, to obtain target semantically enhanced physical attribute features.

[0112] In some implementations, the first acquisition submodule may include a first acquisition unit and a first input unit.

[0113] The first acquisition unit can be used to acquire physical attribute text descriptions, which can be used to characterize the physical attributes of the initial frame image; the first input unit can be used to input the physical attribute text descriptions into the text encoder to obtain the target physical attribute features.

[0114] In some implementations, the second acquisition submodule may be a second acquisition unit, a second input unit, a query embedding unit, and an analysis unit.

[0115] The second acquisition unit can be used to acquire a physical semantic query embedding set, which can be used to characterize the physical application domain of the initial frame image; the second input unit can be used to input the physical semantic query embedding set into the text encoder to obtain the initial physical semantic features; the query embedding unit can be used to perform query embedding on the initial physical semantic features to generate query embedded physical semantic features; the analysis unit can be used to perform principal component analysis on the query embedded physical semantic features to obtain the target physical semantic features.

[0116] In some implementations, the query embedding unit may include a computation subunit.

[0117] The computational subunit can be used to average the first dimension of the initial physical semantic features to obtain the query embedded physical semantic features.

[0118] In some implementations, the second acquisition unit may include an acquisition subunit and an input subunit.

[0119] The acquisition subunit can be used to acquire the text description of the physical application domain; the input subunit can be used to input the text description of the physical application domain into the physical semantic query embedding set generation model to obtain the physical semantic query embedding set. The physical semantic query embedding set generation model can be trained on the large language model based on the sample set labeled with the text description of the physical application domain and the physical semantic query embedding set.

[0120] In some implementations, the perceiver-resampler may include a linear projection layer, a cross-modal attention layer, and a normalization layer connected in sequence.

[0121] The solution provided in this embodiment obtains image features and summary description features corresponding to the initial frame image. The summary description features are used to characterize the physical motion of objects in the initial frame image. It also obtains target semantic-enhanced physical attribute features corresponding to the initial frame image. These target semantic-enhanced physical attribute features characterize the target physical attribute features and target physical semantic features of the initial frame image. The feature space of the target semantic-enhanced physical attribute features is the same as that of the summary description features. These features are then input into a video diffusion model to obtain a sequence of image frames. The sequence of image frames characterizes the physical motion of objects, with the first frame being the initial frame image. Finally, the sequence of image frames is input into a video decoder to obtain the target video. This achieves the prediction of the physical motion of objects in the initial frame image based on the image features, summary description features, and target semantic-enhanced physical attribute features corresponding to the initial frame image, and generates the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs.

[0122] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to in the descriptions of the method embodiments. Any processing method described in the method embodiments can be implemented in the device embodiments through corresponding processing modules, and will not be elaborated upon further in the device embodiments.

[0123] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0124] Please see Figure 5 The diagram illustrates a functional block diagram of an electronic device 500 provided in one embodiment of the present application. The electronic device 500 may include one or more of the following components: a memory 501, a processor 502, and one or more application programs. One or more application programs may be stored in the memory 501 and configured to be executed by one or more processors 502. One or more application programs are configured to perform the methods described in the foregoing method embodiments.

[0125] Memory 501 may include random access memory (RAM) or read-only memory. Memory 501 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 501 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as acquiring image features, acquiring summary description features, acquiring target semantically enhanced physical attribute features, inputting image features, inputting summary description features, inputting target semantically enhanced physical attribute features, obtaining a sequence of image frames, inputting a sequence of image frames, obtaining a target video, acquiring target physical attribute features, acquiring target physical semantic features, mapping target physical attribute features, obtaining initial semantically enhanced physical attribute features, mapping semantically enhanced physical attribute features, obtaining target semantically enhanced physical attribute features, acquiring physical attribute text descriptions, inputting physical attribute text descriptions, obtaining target physical attribute features, acquiring a physical semantic query embedding set, inputting a physical semantic query embedding set, obtaining initial physical semantic features, query embedding, generating query embedding physical semantic features, principal component analysis, obtaining target physical semantic features, averaging calculation, obtaining query embedding physical semantic features, acquiring physical application domain text descriptions, inputting physical application domain text descriptions, obtaining physical semantic query embedding sets, and training a large language model, etc.), and instructions for implementing the various method embodiments described below. The storage data area can also store data created by the electronic device 500 during use (such as initial frame images, image features, summary description features, objects, physical motion, target semantic enhancement physical attribute features, target physical attribute features, target physical semantic features, feature space, video diffusion model, sequence image frames, first frame image, video decoder, target video, resampler-resampler, physical semantic space, initial semantic enhancement physical attribute features, multilayer perceptron, summary feature space, physical attribute text description, physical attribute, text encoder, physical semantic query embedding set, physical application domain, initial physical semantic features, query embedding physical semantic features, physical application domain text description, physical application domain text description labels, physical semantic query embedding set labels, sample set, large language model, linear projection layer, cross-modal attention layer, and normalization layer), etc.

[0126] Processor 502 may include one or more processing cores. Processor 502 connects to various parts within the electronic device 500 using various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing instructions, programs, code sets, or instruction sets stored in memory 501, and by calling data stored in memory 501. Optionally, processor 502 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 502 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 502 and may be implemented separately using a communication chip.

[0127] Please refer to Figure 6 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 600 stores program code 601, which can be called by a processor to execute the methods described in the above method embodiments.

[0128] The computer-readable storage medium 600 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 600 has storage space for program code 601 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 601 may be compressed, for example, in a suitable form.

[0129] Please refer to Figure 7This diagram illustrates a structural block diagram of a computer program product 700 provided in an embodiment of this application. The computer program product 700 includes a computer program / instructions 701, which is stored in a computer-readable storage medium of a computer device. When the computer program product 700 runs on the computer device, the processor of the computer device reads the computer program / instructions 701 from the computer-readable storage medium, and executes the computer program / instructions 701, causing the computer device to perform the methods described in the above method embodiments.

[0130] The solution provided in this embodiment obtains image features and summary description features corresponding to the initial frame image. The summary description features are used to characterize the physical motion of objects in the initial frame image. It also obtains target semantic-enhanced physical attribute features corresponding to the initial frame image. These target semantic-enhanced physical attribute features characterize the target physical attribute features and target physical semantic features of the initial frame image. The feature space of the target semantic-enhanced physical attribute features is the same as that of the summary description features. These features are then input into a video diffusion model to obtain a sequence of image frames. The sequence of image frames characterizes the physical motion of objects, with the first frame being the initial frame image. Finally, the sequence of image frames is input into a video decoder to obtain the target video. This achieves the prediction of the physical motion of objects in the initial frame image based on the image features, summary description features, and target semantic-enhanced physical attribute features corresponding to the initial frame image, and generates the corresponding target video. The image features, summary description features, and target semantic-enhanced physical attribute features enhance the physical attribute expression of the world model in the target video, making the physical motion of objects in the target video more reasonable and improving the matching degree between the generated video and user needs.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A video generation method based on physical property enhancement, characterized in that, include: Obtain image features and summary description features corresponding to the initial frame image, wherein the summary description features are used to characterize the physical motion of objects in the initial frame image; Obtain the target semantic enhanced physical attribute features corresponding to the initial frame image. The target semantic enhanced physical attribute features are used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The target semantic enhanced physical attribute features have the same feature space as the summary description features. The image features, the summary description features, and the target semantic enhancement physical attribute features are input into the video diffusion model to obtain a sequence of image frames. The sequence of image frames is used to characterize the physical motion of the object, and the first frame of the sequence of image frames is the initial frame image. The sequence of image frames is input to the video decoder to obtain the target video.

2. The video generation method according to claim 1, characterized in that, The step of obtaining the target semantic enhancement physical attribute features corresponding to the initial frame image includes: Obtain the target physical property characteristics; Obtain the target physical semantic features; Based on the perceiver-resampler, the target physical attribute features are mapped to the physical semantic space of the target physical semantic features to obtain the initial semantically enhanced physical attribute features; The initial semantically enhanced physical attribute features are mapped to the summary feature space of the summary description features based on the multilayer perceptron to obtain the target semantically enhanced physical attribute features.

3. The video generation method according to claim 2, characterized in that, The process of obtaining the target physical property features includes: Obtain a physical attribute text description, which is used to characterize the physical attributes of the initial frame image; Input the physical attribute text description into the text encoder to obtain the target physical attribute features.

4. The video generation method according to claim 2, characterized in that, The acquisition of the target physical semantic features includes: Obtain a physical semantic query embedding set, which is used to characterize the physical application domain of the initial frame image; Input the physical semantic query embedding set into the text encoder to obtain the initial physical semantic features; The initial physical semantic features are query-embedded to generate query-embedded physical semantic features; Principal component analysis is performed on the embedded physical semantic features of the query to obtain the target physical semantic features.

5. The video generation method according to claim 4, characterized in that, The step of query embedding the initial physical semantic features to generate query embedded physical semantic features includes: The first dimension of the initial physical semantic features is averaged to obtain the query embedded physical semantic features.

6. The video generation method according to claim 4, characterized in that, The process of obtaining the physical semantic query embedding set includes: Obtain textual descriptions of physical application domains; The physical application domain text description is input into the physical semantic query embedding set generation model to obtain the physical semantic query embedding set. The physical semantic query embedding set generation model is obtained by training a large language model based on a sample set labeled with physical application domain text description tags and physical semantic query embedding set tags.

7. The video generation method according to any one of claims 2 to 6, characterized in that, The resampling perceiver-resampler includes a linear projection layer, a cross-modal attention layer, and a normalization layer connected in sequence.

8. A video generation device based on physical property enhancement, characterized in that, include: The first acquisition module is used to acquire image features and summary description features corresponding to the initial frame image, wherein the summary description features are used to characterize the physical motion of objects in the initial frame image; The second acquisition module is used to acquire the target semantic enhanced physical attribute features corresponding to the initial frame image. The target semantic enhanced physical attribute features are used to characterize the target physical attribute features and target physical semantic features of the initial frame image. The target semantic enhanced physical attribute features have the same feature space as the summary description features. The first input module is used to input the image features, the summary description features and the target semantic enhancement physical attribute features into the video diffusion model to obtain a sequence of image frames. The sequence of image frames is used to characterize the physical motion of the object. The first frame of the sequence of image frames is the initial frame image. The second input module is used to input the sequence of image frames to the video decoder to obtain the target video.

9. An electronic device, characterized in that, include: Memory; One or more processors are coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the video generation method as described in any one of claims 1 to 7.

Images