Video generation methods, electronic devices, storage media, and computer program products

By performing image segmentation and material recognition on static images, dynamic effect parameters that conform to physical laws are generated, solving the problem of unreasonable video generation in existing technologies and improving the naturalness and realism of videos.

CN123123656APending Publication Date: 2026-11-17MIGU CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610975729.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies fail to effectively determine the rationality of user prompts during the generation process, resulting in videos that violate physical laws and have poor naturalness and realism.

Method used

By segmenting static images, the material type and physical property information of each image region are identified. Dynamic effect parameters are then generated using the material type and physical property information to simulate dynamic effects that conform to physical laws.

Benefits of technology

It improves the naturalness and realism of generated videos, reduces reliance on complex prompts, and enhances the physical realism of videos and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123656A_ABST
    Figure CN123123656A_ABST
Patent Text Reader

Abstract

This application provides a video generation method, electronic device, storage medium, and computer program product. The method includes: segmenting a first image to obtain one or more image regions; performing material identification on each image region within the one or more image regions to obtain material type information and physical property information corresponding to each image region; generating dynamic effect parameter information corresponding to the first image based on the material type information and physical property information; and generating a video corresponding to the first image based on the dynamic effect parameter information. This method can improve the naturalness and realism of the generated video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video generation method, electronic device, storage medium, and computer program product. Background Technology

[0002] Static images can be used to generate dynamic videos, typically using deep learning models and user-provided text prompts. However, the dynamic videos generated by this method blindly follow the prompts without judging their validity, which may lead to the generated videos violating physical laws and resulting in poor naturalness and realism. Summary of the Invention

[0003] This application provides a video generation method, electronic device, storage medium, and computer program product that can improve the naturalness and realism of the generated video.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a video generation method, the method comprising: Perform image segmentation on the first image to obtain one or more image regions; Material identification is performed on each of the one or more image regions to obtain material type information and physical attribute information corresponding to each image region; Generate dynamic effect parameter information corresponding to the first image based on the material type information and the physical property information; A video corresponding to the first image is generated based on the dynamic effect parameter information.

[0005] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video generation method provided in the embodiments of this application.

[0006] This application provides a storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the video generation method provided in this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the video generation method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: By segmenting the first image, one or more image regions are obtained. Material identification is performed on each of the one or more image regions to obtain the material type information and the corresponding physical attribute information of each image region. The material type information and physical attribute information are used together to determine the dynamic effect parameter information. Based on the dynamic effect parameter information, a video corresponding to the first image is generated. Since the material type information and physical attribute information corresponding to each image region are obtained, the dynamic effect that is more in line with the physical laws can be simulated when generating the video, thereby improving the naturalness and realism of the generated video. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a video generation method provided in an embodiment of this application; Figure 2 A schematic flowchart illustrating an exemplary method for generating video from an image based on material awareness, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] In related technologies, the methods for generating dynamic videos from static images mainly include the following: 1. Methods based on optical flow or motion estimation.

[0014] This method analyzes the displacement of pixels or feature points in an image to calculate the motion field (optical flow), and uses this motion information to generate intermediate frames through interpolation, deformation, and other means, thus forming a video. However, it has limited capabilities in handling complex scenes, non-rigid deformations, and the appearance or disappearance of new objects. The generated dynamics are relatively simple and lack advanced semantics and physical realism.

[0015] 2. Methods based on Generative Adversarial Networks (GANs).

[0016] This method utilizes the adversarial learning mechanism of the generator and discriminator in GANs to learn and generate a series of visually coherent video frames starting from the input static image. Conditional Generative Adversarial Networks (cGANs) are often used for this task, using the input image as the generation condition. However, they often face problems such as training instability and pattern collapse, and struggle to guarantee long-term motion coherence and physical consistency. They lack sufficient understanding of the deep physical properties of image content (such as material and rigidity), potentially generating counterintuitive dynamics.

[0017] 3. Methods based on diffusion models.

[0018] These models generate samples by progressively adding noise to the data (forward process) and learning to recover the data from the noise (backward process). In image-to-video tasks, the input image serves as a condition to guide the backward denoising process, progressively generating a sequence of video frames. While this method excels in generating high-quality and diverse content, it is computationally expensive. More importantly, their adherence to physical laws is more based on imitating patterns in large-scale data rather than explicit physical modeling, thus potentially generating videos that violate physical laws. Furthermore, these two types of deep learning methods (GANs and diffusion models) typically rely heavily on detailed prompts provided by the user to guide the generation, increasing the barrier to entry.

[0019] 4. A method based on image segmentation and specified motion.

[0020] This method first performs semantic or instance segmentation on the input image, dividing it into different objects or regions. Then, it allows the user to manually specify motion paths, animation parameters, or simple dynamic descriptions for these segmented regions, and the system then synthesizes a video based on these instructions. While this enhances controllability, its core flaw lies in the model's inability to understand the inherent properties of the segmented regions. The system blindly executes user-specified motion without judging its suitability for specific materials (such as stone, cloth, or water flow), resulting in stiff, unnatural animations lacking detail and requiring extensive manual adjustments, thus leading to low efficiency.

[0021] Based on this, embodiments of this application provide a video generation method. Figure 1 This is a flowchart illustrating a video generation method provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. Perform image segmentation on the first image to obtain one or more image regions.

[0022] It should be noted that the first image can be understood as a static image of the video to be generated. The first image can be a single image or a sequence of images. The first image can be acquired using an image acquisition model that supports various image formats, such as Joint Photographic Experts Group (JPEG) and Portable Network Graphics (PNG), and can handle images of different resolutions and sizes. Before performing image segmentation on the first image, the method further includes: acquiring a second image, preprocessing the second image, and obtaining the first image. The second image can be understood as the acquired original image, and the first image is the image after preprocessing the original image. Preprocessing steps may include, but are not limited to, scaling, cropping, or color correction. Scaling adjusts the image to a uniform size for easier subsequent processing; cropping removes invalid areas at the image edges, improving processing efficiency; and color correction adjusts the color balance of the image to ensure consistency under different lighting conditions.

[0023] It should be noted that the image segmentation method can be any method capable of segmenting the first image; the specific method can be determined based on the actual situation and is not limited here. As an example, image segmentation methods at least include semantic segmentation based on deep learning, instance segmentation, or texture / edge-based segmentation. Deep learning segmentation uses pre-trained deep learning models (such as Convolutional Networks for Biomedical Image Segmentation (U-Net), Mask Region-based Convolutional Neural Network (Mask R-CNN), etc.) to segment the image and identify different objects or regions (such as people, vehicles, sky, water, vegetation, etc.) in the image. Texture segmentation uses texture analysis methods (such as Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG), etc.) to segment the image and identify regions with similar texture features. Edge detection involves using edge detection algorithms (such as Canny and Sobel) to identify edges in an image and further segment it into different regions. The number of image regions can be one or more, and the specific number can be determined based on the actual situation; no limit is specified here.

[0024] S102. Perform material identification on each image region in one or more image regions to obtain the material type information and the physical attribute information corresponding to the material type information for each image region.

[0025] It should be noted that material type information can be understood as the material type corresponding to each image region, which includes at least cotton, silk, leather, cement, grass, and sand. Physical property information can be understood as the physical properties corresponding to the material type, which includes at least stiffness, flexibility, density, light transmittance, coefficient of friction, and fluid characteristics; physical property information can also be understood as a data structure containing multiple quantified physical parameters. The material recognition method can be any method capable of identifying the material of an image region; the specific method can be determined based on the actual situation and is not limited here.

[0026] S103. Generate dynamic effect parameter information corresponding to the first image based on material type information and physical property information.

[0027] It should be noted that the dynamic effect parameter information can be understood as a dynamic effect parameter set. This parameter set is in time-series format and includes the geometric state information of each image region to be processed at each time (corresponding to each frame of the video). That is, the dynamic effect parameter information can also be understood as frame-by-frame quantized dynamic effect parameters. Generating the dynamic effect parameter information corresponding to the first image based on the material type information and physical property information can be understood as generating the dynamic effect parameters for the corresponding image region based on each image region, the material type information corresponding to each image region, and the physical property information corresponding to the material type information.

[0028] S104. Generate the video corresponding to the first image based on the dynamic effect parameter information.

[0029] It should be noted that the format of the generated video can be determined according to the actual situation and is not limited here; the generated video can be understood as a playable video or a video file, which can be output to a specified storage location or directly pushed to a display device. Generating a video corresponding to the first image based on dynamic effect parameters can be understood as rendering the first image based on dynamic effect parameters to generate a video.

[0030] The solution in this application embodiment obtains the material type information and physical attribute information corresponding to each image region, thereby simulating dynamic effects that are more in line with physical laws when generating videos, thus improving the naturalness and realism of the generated videos.

[0031] In this embodiment, the process of generating dynamic effect parameter information corresponding to the first image based on material type information and physical property information specifically includes: determining the first model corresponding to each image region based on the material type information; and generating dynamic effect parameter information based on the first model and physical property information.

[0032] It should be noted that the first model can be understood as the most commonly used dynamic simulation model selected for each image region based on the material type. The first model includes at least a flexible body dynamics model, a fluid dynamics model, or a rigid body dynamics model; the specific model can be determined according to the actual situation and is not limited here. As an example, if the material type information of an image region is a flexible material (such as "silk" or "hair"), the first model corresponding to that image region can be a flexible body dynamics model; if the material type information of an image region is a fluid material (such as "water surface"), the first model corresponding to that image region can be a fluid dynamics model; if the material type information of an image region is a rigid material (such as "metal"), the first model corresponding to that image region can be a rigid body dynamics model.

[0033] In this embodiment of the application, the process of generating dynamic effect parameter information based on the first model and physical attribute information specifically includes: instantiating the first model based on the physical attribute information to obtain a first model instance; and generating dynamic effect parameter information based on the first model instance and the physical attribute information.

[0034] It should be noted that instantiating the first model based on physical property information can be understood as instantiating the first model for the corresponding image region based on the physical property information corresponding to the material type of each image region; it can also be illustrated as assigning the physical property information to the corresponding variables within the first model to complete the instantiation of the first model, thus obtaining the first model instance. Here, the first model instance can also be understood as the instantiated first model. Generating dynamic effect parameters based on the first model instance and physical property information can be understood as performing iterative simulation calculations on the instantiated first model according to the physical property information at time steps (corresponding to each frame of the video, each frame corresponding to one image) to obtain dynamic effect parameter information; whereby each image region can generate corresponding dynamic effect parameter information based on its corresponding first model instance and corresponding physical property information.

[0035] The solution in this application generates dynamic effect parameters by combining the material type information and physical attribute information corresponding to each image region with a determined first model. This enables the simulation of dynamic effects that are more in line with physical laws when generating videos, thereby improving the naturalness and realism of the generated videos.

[0036] In this embodiment of the application, the method further includes: obtaining prompt word information; and generating dynamic effect parameter information based on the prompt word information, material type information, and physical property information.

[0037] It should be noted that acquiring prompt information can be understood as receiving natural voice prompts from the user, describing the desired dynamic effect, via text or voice. For example, prompt information could include phrases like "a gentle breeze is blowing" or "the character is running." Generating dynamic effect parameters based on prompt information, material type information, and physical property information can be understood as extracting external force information from the prompt information and generating dynamic effect parameters based on this external force information, material type information, and physical property information. Here, external force information can be understood as the core intent or key parameters in the prompt. For example, from "a strong wind is blowing from the left," a force field vector with direction and intensity attributes can be extracted; from "the character is running," a main action command (type: running) can be extracted.

[0038] It should be noted that the process of generating dynamic effect parameters based on external force information, material type information, and physical property information specifically includes: determining a model corresponding to each image region according to the material type, instantiating the first model based on the physical property information to obtain the first model instance, and generating dynamic effect parameters based on the first model instance, external force information, and physical property information.

[0039] It should be noted that a model is determined for each image region based on the material type. The first model is instantiated based on the physical property information. The specific content of the first model instance has been explained in detail above and will not be repeated here. Generating dynamic effect parameters based on the first model instance, external force information, and physical property information can be understood as performing iterative simulation calculations on the instantiated first model according to the external force information and physical property information at time steps (corresponding to each frame of the video) to obtain dynamic effect parameter information. Specifically, each image region can generate corresponding dynamic effect parameter information based on its corresponding first model instance and corresponding physical property information.

[0040] The solution in this application uses the prompt word information as an external input or initial condition in the simulation process, which, together with the acquired physical properties, is used to guide the calculation of dynamic effect parameters. This fusion mechanism ensures the realization of the user's intention while preserving the realism of the physical simulation.

[0041] In this embodiment of the application, the process of performing material identification on each image region in one or more image regions to obtain material type information and physical attribute information corresponding to each image region specifically includes: performing material identification on each image region in one or more image regions using a first analysis method to obtain material type information corresponding to each image region; and determining the physical attribute information corresponding to the material type information based on a physical rule base.

[0042] It should be noted that the first analysis method can be understood as a material analysis method or a material identification method; the specific analysis method can be determined according to the actual situation and is not limited here. As an example, the first analysis method includes at least deep learning-based material classification, texture analysis, and spectral analysis; among them, deep learning material classification: using pre-trained deep learning models (such as Residual Network (ResNet), Contrastive Language–Image Pre-training (CLIP), DENO, etc.) to classify materials and identify the material types (such as cotton, silk, leather, cement, grass, sand, etc.) in different regions of the image. Texture analysis: analyzing the texture features of the image through texture analysis methods (such as gray-level co-occurrence matrix, local binary mode, etc.) to further identify the texture features of different materials. Spectral analysis: using spectral analysis methods (such as spectral reflectance, spectral absorption, etc.) to analyze the spectral features of the image to further identify the spectral features of different materials.

[0043] In this embodiment of the application, the process of determining the physical attribute information corresponding to the material type based on the physical rule base specifically includes: using the material type information as an index to query the physical rule base and obtain the physical attribute information.

[0044] It should be noted that the physical rule base can be understood as a database that has been established and is used to store the physical properties of different materials. Physical property information can also be understood as a data structure that includes multiple quantified physical parameters. Using material type information as an index to query the physical rule base and obtain physical property information can be understood as using the material type tag as an index to query the physical rule base and retrieve the physical property information corresponding to that material.

[0045] In this embodiment of the application, the process of generating a video corresponding to the first image based on dynamic effect parameter information specifically includes: rendering the first image using dynamic effect parameter information to generate a video.

[0046] It should be noted that rendering the first image using dynamic effect parameters to generate a video can be understood as using dynamic effect parameters to perform geometric deformation and image rendering on each image region in the first image to obtain a processed image, and then combining the processed images in chronological order to obtain a video.

[0047] For ease of understanding, an example is provided here. The first image includes various materials (such as silk and water). The physical property of silk is at least that it can float, and the physical property of water is at least that it can ripple. However, the amplitude of floating and ripple is not the same. The solution of this application generates frame-by-frame dynamic effect parameters corresponding to the corresponding regions based on the physical property information of the materials (such as the dynamic effect parameters of the silk in the image region during the floating process in each frame (or time step, such as the 1st second, the 2nd second, and the 3rd second, etc.), and the dynamic effect parameters of the horizontal in the image region during the waveguide ripple process in each frame (or time step, such as the 1st second, the 2nd second, and the 3rd second, etc.), so that the generated video can conform to their respective physical properties.

[0048] The solution in this application embodiment deeply understands the semantic information and physical properties of image content, especially the material information of key areas, and deeply couples material recognition with physical properties to guide and constrain the process of generating dynamic videos from single or a small number of static images. This allows for the simulation of dynamic effects that are more in line with physical laws when generating videos, reducing reliance on complex prompts and improving the naturalness, realism, and user experience of the generated videos.

[0049] For ease of understanding, the video generation method described above can be understood as a method for generating videos from images based on material awareness. The specific steps are as follows.

[0050] 1. Image acquisition and preprocessing.

[0051] Image Acquisition Module: This module is responsible for acquiring the input static images. The input image can be a single image or a sequence of images used to generate the video. The image acquisition module supports various image formats, such as JPEG and PNG, and can handle images of different resolutions and sizes.

[0052] Preprocessing: After acquiring the image, necessary preprocessing operations are performed, such as scaling, cropping, and color correction, to ensure the image meets the requirements of subsequent processing. Preprocessing steps may include, but are not limited to: Scaling: Adjusting the image to a uniform size for easier subsequent processing. Cropping: Removing invalid areas from the image edges to improve processing efficiency. Color Correction: Adjusting the color balance of the image to ensure consistency under different lighting conditions.

[0053] 2. Image segmentation and material recognition.

[0054] Image Segmentation Module: This module is responsible for segmenting the input static image to obtain at least one image region. Image segmentation can employ various methods, such as semantic segmentation based on deep learning, instance segmentation, or texture / edge-based segmentation methods. The specific steps are as follows: Deep Learning Segmentation: Using pre-trained deep learning models (such as U-Net, Mask R-CNN, etc.) for image segmentation, identifying different objects or regions in the image (such as people, vehicles, sky, water, vegetation, etc.). Texture Segmentation: Utilizing texture analysis methods (such as SIFT, HOG, etc.) to segment the image, identifying regions with similar texture features. Edge Detection: Identifying edges in the image using edge detection algorithms (such as Canny, Sobel, etc.), further segmenting different regions.

[0055] Material Recognition Module: This module is responsible for identifying the material of the segmented image regions, determining the material type and / or related physical properties of the image regions. Material recognition can employ various methods, such as deep learning-based material classification, texture analysis, and spectral analysis. The specific steps are as follows: Deep Learning Material Classification: Using pre-trained deep learning models (such as ResNet, CLIP, DENO, etc.), material classification is performed to identify the material types (such as cotton, silk, leather, cement, grass, sand, etc.) of different regions in the image. Texture Analysis: Texture analysis methods (such as gray-level co-occurrence matrix, local binary mode, etc.) are used to analyze the texture features of the image, further identifying the texture features of different materials. Spectral Analysis: Spectral analysis methods (such as spectral reflectance, spectral absorption, etc.) are used to analyze the spectral features of the image, further identifying the spectral features of different materials.

[0056] 3. Dynamic effects generation.

[0057] This module is the core calculation unit for dynamic effects. It receives a specified image region and its material type, and through structured physical simulation calculations, generates a set of precise, quantified, frame-by-frame dynamic effect parameters for that region, which can be used for video frame compositing. The generation of dynamic effect parameters follows a rigorous and quantifiable calculation workflow, specifically including the following four steps.

[0058] (1) Input acquisition and initialization.

[0059] This module obtains the initial information required for calculation from other modules: Image region information: One or more image regions to be processed, output from the image segmentation module in step 2 above.

[0060] Material type label: The material type (e.g., "silk", "water surface", "metal") determined by the material identification module in step 2 above for each of the above areas.

[0061] (Optional) External force information: External forces obtained from the user interface module in step 5 below and parsed from the user prompt words (e.g., "a breeze is blowing"), or global forces preset by the system (e.g., gravity).

[0062] (2) Physical attribute vector retrieval.

[0063] A physical rule base is established to store the physical properties of different materials (such as stiffness, flexibility, density, light transmittance, coefficient of friction, fluid characteristics, etc.). The system uses material type tags as indexes to query the internal physical rule base and retrieve the "basic physical property vector" that strictly corresponds to that material. This is a data structure containing multiple quantified physical parameters. For example: The vector V_silk corresponding to the "silk" material may be: {mass per area: 0.1; stiffness: 5.0; damping factor: 0.95}.

[0064] The vector V_water corresponding to the "water surface" material might be: {density: 1000; viscosity: 0.001; surface tension: 0.072}.

[0065] The vector V_metal corresponding to the "metal" material might be: {density: 7850; stiffness: 1.0; restitution coefficient: 0.4}.

[0066] (3) Selection and instantiation of dynamic simulation models.

[0067] The system automatically selects the most suitable dynamic simulation model for each region based on the material type. For example, if the material is flexible (such as "silk" or "hair"), a flexible body dynamics model is selected; if the material is fluid (such as "water surface"), a fluid dynamics model is selected; and if the material is rigid (such as "metal"), a rigid body dynamics model is selected.

[0068] After selecting a model, the system assigns the physical attribute vector retrieved in step (2) above to the corresponding variable inside the model, completing the instantiation of the model and preparing for the next step of calculation.

[0069] (4) Iterative calculation and parameter generation.

[0070] The instantiated model will perform iterative simulation calculations at time steps (corresponding to each frame of the video) based on external force information (optional) (such as "breeze" parsed from the user prompt or the default gravity) and internal physical properties. The following is an example of iterative simulation calculations based on external force information and physical property information.

[0071] Example of flexible body parameter calculation ("Silk" under "Breeze").

[0072] Simulation model: Mass-spring system (a flexible body dynamics model).

[0073] Calculation process: For each frame of the video, the model performs the following calculations for each vertex in the mesh representing silk: (a) Calculate the internal spring force generated by the connected vertices based on the spring constant (using Hooke's law).

[0074] (b) Calculate the internal damping force based on the damping coefficient and the current velocity at the apex.

[0075] (c) The external force generated by the "breeze" is superimposed.

[0076] (d) Calculate the acceleration of the vertex according to Newton's second law (F=ma) and the mass per unit area.

[0077] (e) Update the velocity and three-dimensional position of the vertex by numerical integration (such as Euler method).

[0078] Output parameters: The final output is a time series containing the three-dimensional coordinate positions of all vertices in each frame, which is the dynamic parameter that drives the silk to move.

[0079] Example of fluid parameter calculation (ripples on the "water surface").

[0080] Simulation model: a solver based on the wave equation (a fluid dynamics model).

[0081] Calculation process: For each frame of the video, the model performs the following calculations on the two-dimensional grid representing the water surface: (a) Apply an initial perturbation (such as simulating an object falling into water).

[0082] (b) Calculate the wave propagation speed based on density and surface tension. (c) According to the wave equation ( ²y / t² = c² (²y), calculate the vertical height of each point in the grid at the next time step.

[0083] (d) Viscosity parameters are used to calculate the energy attenuation of waves during propagation.

[0084] Output parameters: The final output is a time series containing a 2D height map of each frame. The video compositing module will use this sequence to perform vertex displacement on the water surface mesh, creating ripples.

[0085] 4. Video merging and output.

[0086] This module takes the frame-by-frame quantized motion effect parameters output by the motion effect generation module in step 3, and renders and encodes them to generate the final video file. The module's workflow is shown below.

[0087] (1) Input data reception: The module receives the dynamic effect parameter set output by the dynamic effect generation module. The parameter set is in time series format and contains the geometric state information of each image region to be processed in each frame of the video (such as vertex coordinate sequence, height field data, etc.).

[0088] (2) Frame Generation: This process iterates through video frames, generating a corresponding image for each time step in the parameter set. For the nth frame of the video, the rendering process is as follows: (a) Parameter application: Extract the data corresponding to time step t_n from the parameter set.

[0089] (b) Geometric Transformation: The parameters of this frame are applied to the corresponding segmented regions of the original static image to perform geometric transformations. For example, a sequence of vertex coordinates is applied to a mesh of a region to achieve dynamic deformation of that region.

[0090] (c) Image Rendering: Perform shading and lighting calculations on the transformed geometry and scene to generate a complete pixel image of the frame. The rendering method can be configured according to requirements, including but not limited to physically based rendering (PBR) or deep learning-based rendering models.

[0091] (3) Video sequence synthesis: After all target frames have been rendered, these independent frame images are combined in chronological order.

[0092] (a) Basic composition: Arrange the frames in the order of t_1, t_2, ..., t_n to form a complete video sequence.

[0093] (b) Optional enhancements: Optionally, to improve video smoothness or generate slow-motion effects, inter-frame interpolation-based compositing techniques can be used between rendered frames to generate new intermediate frames.

[0094] 5. User interface module (optional).

[0095] This module provides an interactive interface for users, responsible for receiving, parsing, and transmitting text prompts input by the user to provide macro-level guidance for the generation of dynamic effects. The module's workflow is defined as follows: (1) Prompt input: The module provides an input interface that allows users to input natural language prompts describing the desired dynamic effect (e.g., “a breeze is blowing”, “the character is running”) via text or voice.

[0096] (2) Prompt word parsing: The received natural language prompt words are sent to an internal parser, which is responsible for extracting the core intent or key parameters in the prompt words. For example, from "A strong wind blows from the left", a force field vector with direction and intensity attributes is parsed out; from "The person runs", a main action instruction (type: running) is parsed out.

[0097] (3) Information Fusion and Transmission: The parsed structured information (such as force field vectors and action commands) is transmitted to the dynamic effect generation module in step 3. This information does not directly determine the final dynamic details, but serves as an external input or initial condition in the physical simulation process. It will be used together with the physical properties identified from the materials to guide the calculation of dynamic effect parameters. This fusion mechanism ensures the realization of the user's intentions while preserving the realism of the physical simulation.

[0098] For ease of understanding, Figure 2 A flowchart illustrating an exemplary method for generating video from images based on material awareness, as provided in this application embodiment, is shown below. Figure 2 As shown, the specific steps are as follows: 1. Obtain static images and preprocess them.

[0099] 2. Image segmentation module.

[0100] It should be noted that after the image segmentation model segments the region, it is input into the material recognition module.

[0101] 3. Material recognition module.

[0102] It should be noted that the material recognition model outputs the material type.

[0103] 4. User interface module (optional).

[0104] It should be noted that the user interface module outputs external force.

[0105] 5. Input: Material type & external force.

[0106] It should be noted that external force is optional information.

[0107] 6. Retrieve physical attribute vectors from the physical rule base.

[0108] 7. Select a dynamic simulation model.

[0109] It should be noted that if the material type is flexible, proceed to step 8; if the material type is fluid, proceed to step 9; if the material type is rigid, proceed to step 10.

[0110] 8. Flexible body dynamics model.

[0111] 9. Fluid dynamics model.

[0112] 10. Rigid body dynamics model.

[0113] 11. Iteratively calculate and generate frame-by-frame dynamic parameters.

[0114] It should be noted that steps 3-11 are the steps executed by the dynamic effects generation module.

[0115] 12. Video synthesis module.

[0116] 13. Generate images frame by frame.

[0117] It should be noted that the video compositing module renders images frame by frame based on the parameters.

[0118] 14. Combined into a video sequence.

[0119] It should be noted that steps 12-14 are video synthesis and output.

[0120] 15. End / Output Video.

[0121] This application provides a video generation apparatus. Figure 3 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application; as shown below. Figure 3 As shown, the video generation device 300 includes: Image segmentation unit 301 is used to segment the first image to obtain one or more image regions; The material recognition unit 302 is used to perform material recognition on each image region in the one or more image regions to obtain material type information and physical attribute information corresponding to each image region. The first generation unit 303 is used to generate dynamic effect parameter information corresponding to the first image based on the material type information and the physical property information. The second generation unit 304 is used to generate a video corresponding to the first image based on the dynamic effect parameter information.

[0122] In one embodiment, the first generation unit 303 is further configured to determine a first model corresponding to each image region based on the material type information; and generate the dynamic effect parameter information based on the first model and the physical property information.

[0123] In one embodiment, the first generation unit 303 is further configured to instantiate the first model based on the physical attribute information to obtain a first model instance; and generate the dynamic effect parameter information based on the first model instance and the physical attribute information.

[0124] In one embodiment, the video generation apparatus 300 further includes an acquisition unit and a third generation unit; The acquisition unit is used to acquire prompt word information; The third generation unit is used to generate the dynamic effect parameter information based on the prompt word information, the material type information, and the physical property information.

[0125] In one embodiment, the material identification unit 302 is further configured to perform material identification on each image region in the one or more image regions using a first analysis method to obtain material type information corresponding to each image region; and determine the physical attribute information corresponding to the material type information based on a physical rule base.

[0126] In one embodiment, the material identification unit 302 is further configured to use the material type information as an index to query the physical rule base to obtain the physical attribute information.

[0127] In one embodiment, the second generation unit 304 is further configured to render the first image using the dynamic effect parameter information to generate the video.

[0128] This application also provides an electronic device. Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 4 As shown, the electronic device 400 includes a processor 401 and a memory 403. Optionally, the electronic device 400 may also include a communication bus 402.

[0129] In specific embodiments, the processor 401 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not impose specific limitations.

[0130] In this embodiment, the communication bus 402 is used to establish a connection and communication between the processor 401 and the memory 403; when the processor 401 executes the running program stored in the memory 403, it implements the following video generation method: The first image is segmented to obtain one or more image regions; material identification is performed on each of the one or more image regions to obtain material type information and physical attribute information corresponding to each image region; dynamic effect parameter information corresponding to the first image is generated based on the material type information and the physical attribute information; and a video corresponding to the first image is generated based on the dynamic effect parameter information.

[0131] Furthermore, the processor 401 is also configured to determine the first model corresponding to each image region based on the material type information; and generate the dynamic effect parameter information based on the first model and the physical property information.

[0132] Furthermore, the processor 401 is also used to instantiate the first model based on the physical attribute information to obtain a first model instance; and to generate the dynamic effect parameter information based on the first model instance and the physical attribute information.

[0133] Furthermore, the processor 401 is also used to acquire prompt word information; and to generate the dynamic effect parameter information based on the prompt word information, the material type information, and the physical property information.

[0134] Furthermore, the processor 401 is also used to perform material identification on each image region in the one or more image regions using the first analysis method to obtain material type information corresponding to each image region; and to determine the physical attribute information corresponding to the material type information based on the physical rule base.

[0135] Furthermore, the processor 401 is also used to use the material type information as an index to query the physical rule base to obtain the physical attribute information.

[0136] Furthermore, the processor 401 is also used to render the first image using the dynamic effect parameter information to generate the video.

[0137] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the video generation method described above.

[0138] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the video generation method described above.

[0139] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0141] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of video generation, the method comprising: The method includes: Perform image segmentation on the first image to obtain one or more image regions; Material identification is performed on each of the one or more image regions to obtain material type information and physical attribute information corresponding to each image region; Generate dynamic effect parameter information corresponding to the first image based on the material type information and the physical property information; A video corresponding to the first image is generated based on the dynamic effect parameter information.

2. The method of claim 1, wherein, The step of generating dynamic effect parameter information corresponding to the first image based on the material type information and the physical property information includes: The first model corresponding to each image region is determined based on the material type information; The dynamic effect parameter information is generated based on the first model and the physical property information.

3. The method of claim 2, wherein, The generation of the dynamic effect parameter information based on the first model and the physical property information includes: The first model is instantiated based on the physical property information to obtain a first model instance; The dynamic effect parameter information is generated based on the first model instance and the physical attribute information.

4. The method of claim 1, wherein, The method further includes: Obtain prompt word information; The dynamic effect parameter information is generated based on the prompt word information, the material type information, and the physical property information.

5. The method of claim 1, wherein, The step of performing material identification on each of the one or more image regions to obtain material type information and physical attribute information corresponding to each image region includes: The material type information corresponding to each of the one or more image regions is obtained by using the first analysis method to identify the material of each image region. The physical attribute information corresponding to the material type information is determined based on the physical rule base.

6. The method of claim 5, wherein, The step of determining the physical attribute information corresponding to the material type information based on the physical rule base includes: The material type information is used as an index to query the physical rule base and obtain the physical attribute information.

7. The method of claim 1, wherein, The step of generating the video corresponding to the first image based on the dynamic effect parameter information includes: The first image is rendered using the dynamic effect parameter information to generate the video.

8. An electronic device, comprising: The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A storage medium having stored thereon a computer program or computer- executable instructions, characterized in that, When the computer program or the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or the computer-executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.