Animation generation method and device, electronic equipment and storage medium
By introducing a target motion simulation model and temporal attention calculation into the image generation model, the problem of inconsistent physical laws in animation generation is solved, and simulation animation generation with high consistency and accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-06
- Publication Date
- 2026-04-21
AI Technical Summary
Existing image generation models lack explicit modeling of physical laws, resulting in generated animations that do not accurately conform to physical laws, or low physical consistency between different frames, which affects the accuracy of physical simulation animations.
By acquiring initial image features and motion description text, the pixel coordinate displacement is inferred using the target motion simulation model. Combined with physical image features and temporal attention calculation, a simulation animation that conforms to physical laws is generated.
It improves the timing consistency and physical conformity of the animation, ensuring the accuracy of the generated animation.
Smart Images
Figure CN121904237A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of medical technology and financial technology, and particularly to an animation generation method and apparatus, electronic device and storage medium. Background Technology
[0002] Currently, users can input static images and descriptions of desired image motion into image generation models, enabling the models to generate continuous animations to simulate corresponding dynamic processes. For example, in the medical field, simulating surgical robots involves inputting an image before a desired operation and a detailed description of the desired motion. The model then outputs a simulation animation of the surgical robot performing the desired operation, assisting medical personnel in adjusting surgical procedures. Similarly, in the financial field, in insurance claims, users can input static images of insured property, the property's state, and a detailed description of the accident (e.g., a high-speed collision between an insured vehicle and an oncoming vehicle). The model generates a simulation animation of property damage reflecting the accident, assisting in claims assessment. However, current image generation models lack explicit modeling of physical laws, resulting in animations that don't accurately conform to physical laws or exhibit low physical consistency across frames, leading to low accuracy in physical simulation animations. Therefore, generating accurate physical simulation animations has become a pressing technical problem. Summary of the Invention
[0003] The main objective of this application is to provide an animation generation method, apparatus, electronic device, and storage medium, aiming to improve the accuracy of animation generation.
[0004] To achieve the above objectives, a first aspect of this application proposes an animation generation method, the method comprising: An initial image is acquired, and image features are extracted from the initial image to obtain the initial image features; Obtain motion description text, and use a preset target motion simulation model to infer coordinate displacement of the motion description text and the initial image features to obtain the pixel coordinate displacement of each pixel in the initial image. Based on the initial image features and the pixel coordinate displacement, feature map sampling is performed to obtain physical image features; Temporal attention features are obtained by performing temporal attention calculation on the physical image features and the initial image features. Image generation is performed based on the initial image features, the physical image features, and the temporal attention features to obtain candidate images; An ordered sequence is constructed based on the initial image and the candidate images to obtain the target simulation animation.
[0005] In some embodiments, before performing coordinate displacement inference on the motion description text and the initial image features using a preset target motion simulation model, the method further includes pre-training the target motion simulation model, specifically including: Obtain the initial image of the sample, and extract image features from the initial image of the sample to obtain the sample image features; The sample description text is obtained, and the initial velocity inference submodule of the preset initial motion simulation model is used to perform velocity inference on the sample image features and the sample description text to obtain the predicted velocity data of each pixel in the sample starting image. The initial displacement inference submodule of the initial motion simulation model performs displacement calculation on the predicted velocity data to obtain the reference coordinate displacement of each pixel in the initial image of the sample. Based on the predicted velocity data, the velocity curl term and velocity divergence term are calculated, and based on the reference coordinate displacement, the collision penalty term is calculated. The initial motion simulation model is adjusted by means of the velocity curl term, the velocity divergence term, and the calculated collision penalty term to obtain the target motion simulation model.
[0006] In some embodiments, the step of calculating the velocity curl and velocity divergence terms based on the predicted velocity data, and calculating the collision penalty term based on the reference coordinate displacement, includes: The curl of the predicted velocity data is calculated to obtain the velocity curl; The curl norm is obtained by calculating the Euclidean norm of the velocity curl. The predicted velocity data is subjected to divergence calculation to obtain the velocity divergence; The Euclidean norm of the velocity divergence is calculated to obtain the divergence norm. The displacement of the reference coordinates and the preset collision position are used to calculate the collision displacement offset. The collision displacement offset is calculated using the Manhattan norm to obtain the collision displacement norm; The expected value of the curl norm is used as the velocity curl term, the expected value of the divergence norm is used as the velocity divergence term, and the expected value of the collision displacement norm is used as the collision penalty term.
[0007] In some embodiments, the step of inferring the coordinate displacement of the motion description text and the initial image features using a preset target motion simulation model to obtain the pixel coordinate displacement of each pixel in the initial image includes: Obtain motion parameters from the motion description text; The target velocity inference submodule of the target motion simulation model performs velocity prediction on the motion parameters and the initial image features to obtain pixel velocity data for each pixel in the initial image. The target displacement inference submodule of the target motion simulation model calculates the pixel coordinate displacement by using the pixel velocity data and the preset inter-frame time period.
[0008] In some embodiments, the step of using the target velocity inference submodule of the target motion simulation model to predict the velocity of the motion parameters and the initial image features to obtain pixel velocity data for each pixel in the initial image includes: The initial image features are subjected to dimensionality transformation to obtain intermediate image features; The Hadamard product is calculated based on the intermediate image features and the motion parameters to obtain the pixel velocity data.
[0009] In some embodiments, the step of performing temporal attention calculation on the physical image features and the initial image features to obtain temporal attention features includes: Linear mapping is performed on the physical image features to obtain the predicted key vector and the predicted value vector; The initial image features are linearly mapped to obtain the initial query vector; The temporal attention weights are obtained by calculating the weights based on the predicted key vector and the initial query vector. The temporal attention features are obtained by weighted summation of the temporal attention weights and the predicted value vector.
[0010] In some embodiments, the step of generating candidate images based on the initial image features, the physical image features, and the temporal attention features includes: The initial image features are convolved to obtain convolved image features; The convolutional image features are activated to obtain intermediate image features; The intermediate image features, the physical image features, and the temporal attention features are fused to obtain the fused image features; The fused image features are used to construct the candidate image.
[0011] To achieve the above objectives, a second aspect of this application provides an animation generation apparatus, the apparatus comprising: The image feature extraction module is used to acquire the initial image and extract image features from the initial image to obtain the initial image features; The displacement inference module is used to acquire motion description text and perform coordinate displacement inference on motion description text and initial image features through a preset target motion simulation model to obtain the pixel coordinate displacement of each pixel in the initial image. The physical feature sampling module is used to sample the feature map based on the initial image features and pixel coordinate displacement to obtain physical image features. The temporal attention module is used to perform temporal attention calculations on physical image features and initial image features to obtain temporal attention features. The image generation module is used to generate candidate images based on initial image features, physical image features, and temporal attention features. The animation construction module is used to construct an ordered sequence based on the initial image and candidate images to obtain the target simulation animation.
[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.
[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.
[0014] The animation generation method, apparatus, electronic device, and storage medium proposed in this application extract initial image features from the initial image. These features, along with motion description text, are then input into a target motion simulation model, which infers the pixel coordinate displacement of each pixel in the initial image. Next, feature map sampling is performed on the displacement and the initial image features to obtain physical image features, ensuring that subsequently generated images conform to physical laws. Then, temporal attention features are generated by performing temporal attention calculations on the initial image features and physical image features, effectively capturing dynamic changes and spatial dependencies between adjacent frames and ensuring high consistency between consecutive frames. Finally, the generated candidate images are integrated into an ordered sequence to construct the target simulation animation. The embodiments of this application not only improve the temporal consistency of the animation but also ensure that the displayed content conforms to physical laws, ultimately improving the accuracy of animation generation. Attached Figure Description
[0015] Figure 1 This is a flowchart of the animation generation method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the training process of the target simulation model provided in this application embodiment; Figure 3 yes Figure 1 The flowchart of step S102 in the document; Figure 4 yes Figure 1 The flowchart of step S104 in the process; Figure 5 yes Figure 1 The flowchart of step S105 in the process; Figure 6 This is a schematic diagram of the structure of the animation generation device provided in the embodiments of this application; Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0019] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.
[0020] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.
[0021] Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events from natural language text and outputs it as structured data. Information extraction is a technique for extracting specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of smaller, specific units, such as characters, words, phrases, sentences, paragraphs, or combinations of these units. Extracting noun phrases, names of people, and place names from text data is an example of text information extraction. Of course, text information extraction techniques can extract information of various types.
[0022] Currently, users can input static images and descriptions of desired image motion into image generation models, enabling the models to generate continuous animations to simulate corresponding dynamic processes. For example, in the medical field, simulating surgical robots involves inputting an image before a desired operation and a detailed description of the desired motion. The model then outputs a simulation animation of the surgical robot performing the desired operation, assisting medical personnel in adjusting surgical procedures. Similarly, in the financial field, in insurance claims, users can input static images of insured property, the property's state, and a detailed description of the accident (e.g., a high-speed collision between an insured vehicle and an oncoming vehicle). The model generates a simulation animation of property damage reflecting the accident, assisting in claims assessment. However, current image generation models lack explicit modeling of physical laws, resulting in animations that don't accurately conform to physical laws or exhibit low physical consistency across frames, leading to low accuracy in physical simulation animations. Therefore, generating accurate physical simulation animations has become a pressing technical problem.
[0023] Based on this, embodiments of this application provide an animation generation method and apparatus, electronic device and storage medium, aiming to improve the accuracy of animation generation.
[0024] The animation generation method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the animation generation method in this application is described.
[0025] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0026] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0027] The animation generation method provided in this application relates to the field of artificial intelligence technology. The animation generation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the animation generation method, but is not limited to the above forms.
[0028] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0029] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0030] Figure 1 This is an optional flowchart of the animation generation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.
[0031] Step S101: Obtain the initial image and extract image features from the initial image to obtain the initial image features.
[0032] Step S102: Obtain motion description text, and use a preset target motion simulation model to infer coordinate displacement of the motion description text and initial image features to obtain the pixel coordinate displacement of each pixel in the initial image.
[0033] Step S103: Based on the initial image features and pixel coordinate displacement, feature map sampling is performed to obtain physical image features.
[0034] Step S104: Perform temporal attention calculation on the physical image features and the initial image features to obtain temporal attention features.
[0035] Step S105: Generate candidate images based on initial image features, physical image features, and temporal attention features.
[0036] Step S106: Construct an ordered sequence based on the initial image and candidate images to obtain the target simulation animation.
[0037] Steps S101 to S106 of this embodiment involve extracting initial image features from the initial image, then inputting these features and motion description text into the target motion simulation model. This model infers the pixel coordinate displacement of each pixel in the initial image. Next, feature map sampling is performed on the displacement and the initial image features to obtain physical image features, ensuring that subsequently generated images conform to physical laws. Then, temporal attention features are generated by performing temporal attention calculations on the initial image features and physical image features. This effectively captures dynamic changes and spatial dependencies between adjacent frames, ensuring high consistency between consecutive frames. Finally, the generated candidate images are integrated into an ordered sequence to construct the target simulation animation. This embodiment not only improves the temporal consistency of the animation but also ensures that the displayed content conforms to physical laws, ultimately improving the accuracy of animation generation.
[0038] In step S101 of some embodiments, the initial image refers to an image used to describe a scene or object before generating the animation; it is typically a static image input by the user. In a medical machinery motion simulation scenario, the initial image could be a top view of the surgical robot and the operating table.
[0039] In some embodiments, a convolutional neural network can be used to process the initial image and extract high-level features, namely the initial image features. For example, for an image of a surgical robot, a convolutional neural network can extract information such as the robot's edges, structure, joints, and end effector position, and convert them into feature vectors.
[0040] In some embodiments, prior to step S102, the animation generation method further includes pre-training a target motion simulation model, which is used to output the pixel coordinate displacement of each pixel based on the image. Preferably, the output of the target motion simulation model is output in the form of a displacement field. Where H is the height of the initial image, W is the width of the initial image, and 2 indicates that the displacement vector of each pixel has two components: one is the displacement in the x-direction and the other is the displacement in the y-direction.
[0041] Please see Figure 2 Step S102 may include, but is not limited to, steps S201 to S205: Step S201: Obtain the initial image of the sample and extract image features from the initial image of the sample to obtain the sample image features.
[0042] Step S202: Obtain sample description text, and perform velocity inference on sample image features and sample description text through the initial velocity inference submodule of the preset initial motion simulation model to obtain the predicted velocity data of each pixel in the sample starting image.
[0043] Step S203: The displacement calculation of the predicted velocity data is performed by the initial displacement inference submodule of the initial motion simulation model to obtain the reference coordinate displacement of each pixel in the initial image of the sample.
[0044] Step S204: Calculate the velocity curl term and velocity divergence term based on the predicted velocity data, and calculate the collision penalty term based on the reference coordinate displacement.
[0045] Step S205: Adjust the parameters of the initial motion simulation model based on the velocity curl term, velocity divergence term, and collision penalty term to obtain the target motion simulation model.
[0046] In step S201 of some embodiments, the initial sample image refers to a static sample image used for training. The method of image feature extraction is consistent with the specific embodiment of step S101, and will not be described again here.
[0047] In step S202 of some embodiments, the sample description text refers to a textual description of the motion trajectory, action, or state of the target object or scene.
[0048] In some embodiments, natural language processing (NLP) technology can be used to parse the sample description text to obtain the corresponding motion parameters. For example, in a medical technology application scenario, the sample description text could be: "The surgical robot's end effector is currently stationary, and then performs a uniform linear cut along the negative y-axis at a speed of 3 millimeters per second, with the cutting area being the skin surface." The initial velocity inference submodule can perform text analysis on the sample description text to obtain the end effector's current initial velocity as 0 mm / s, its motion speed as 3 mm / s, and the coefficient of friction between the scalpel and the skin. In a financial insurance claims assessment scenario, the sample description text could be: "The insured vehicle is currently traveling along the road at a speed of 40 km / h, and then scrapes against a car parked on the side of the road." The initial velocity inference submodule can perform text analysis on the sample description text to obtain the end effector's current initial velocity as 40 km / h and the coefficient of friction, etc.
[0049] The initial velocity inference submodule of the initial motion simulation model predicts the velocity of each pixel in the initial sample image, obtaining the velocity data for each pixel, which is the predicted velocity data. Specifically, the initial velocity inference submodule also includes a multilayer perceptron (MLP) structure, which receives the features of the sample image and then performs a Hadamard product with the previously obtained motion parameters. The process of the initial velocity inference submodule inferring the velocity can be referred to in the following analytical expression: (1), in, This represents the predicted velocity data for the current frame. In this specific embodiment, the current frame is the initial sample image. θ represents the parameters of the MLP network. z represents the features of the sample image. This represents the motion parameters of the current frame. This represents the Hadamard product operation.
[0050] In step S203 of some embodiments, the initial displacement inference submodule determines the kinematic model corresponding to the motion description and calculates the displacement of each pixel using the predicted velocity data. Taking a physics model of uniformly accelerated motion as an example, the process of the initial displacement inference submodule calculating the displacement based on the predicted velocity data can be referred to the following analytical expression: (2), in, This indicates the displacement of the current frame, i.e., the displacement of the reference coordinates. This indicates the duration between animation frames, and is usually a preset value. It represents acceleration. This represents the predicted velocity data for the current frame.
[0051] In other embodiments, if the moving object described in the sample description text is a fluid, the corresponding fluid kinematics model can be determined for displacement inference, but this is not limited to this.
[0052] In step S204 of some embodiments, the calculation order of the velocity curl term, velocity divergence term, and collision penalty term is not strictly limited. The calculation process can refer to the following steps: 1. Calculate the curl of the predicted velocity data to obtain the velocity curl. 2. Calculate the Euclidean norm of the velocity curl to obtain the curl norm. In this embodiment, after obtaining the Euclidean norm corresponding to the velocity curl, the Euclidean norm is squared, and the squared result is used as the curl norm. 3. Calculate the divergence of the predicted velocity data to obtain the velocity divergence. 4. Calculate the Euclidean norm of the velocity divergence to obtain the divergence norm. In this embodiment, after obtaining the Euclidean norm corresponding to the velocity divergence, the Euclidean norm is squared, and the squared result is used as the divergence norm. 5. Calculate the displacement of the reference coordinate displacement and the preset collision position to obtain the collision displacement offset. It should be noted that the collision location refers to the area in an image where an object contacts or intersects with other objects; this is typically the part that requires special handling in physical simulations. The definition of the collision location is not limited to the outer boundary of the object; it can also include any point where objects come into contact with each other between different time steps. The collision displacement offset refers to the displacement amount by which an object in the image, moving according to the reference coordinate displacement, will penetrate or overlap with other objects. 6. Calculate the Manhattan norm of the collision displacement offset to obtain the collision displacement norm. 7. Use the expected value of the curl norm as the velocity curl term, the expected value of the divergence norm as the velocity divergence term, and the expected value of the collision displacement norm as the collision penalty term.
[0053] In step S205 of some embodiments, a loss function is constructed by combining the velocity curl term, the velocity divergence term, and the collision penalty term. This loss function can be referred to in the following analytical expression: (3), in, This represents the velocity curl term. This represents the velocity divergence term. This represents the collision penalty term. The parameters of the initial motion simulation model are adjusted until the loss function value is zero, thus obtaining the target motion simulation model.
[0054] Steps S201 to S205, as illustrated in this embodiment, involve acquiring the initial sample image, extracting its features, and inferring the velocity data of each pixel by combining it with the sample description text. Next, the displacement inference submodule of the initial motion simulation model calculates the displacement of the predicted velocity data to obtain the reference coordinate displacement of each pixel. Further calculations are made of the curl and divergence terms of the velocity, and a collision penalty term is calculated based on the reference coordinate displacement, effectively ensuring physical consistency and accuracy during the simulation process. Finally, these calculations are used to adjust the parameters of the initial motion simulation model, resulting in an optimized target motion simulation model. This completes the training of the initial motion simulation model, further ensuring the physical accuracy and temporal consistency of the simulation animation.
[0055] In step S102 of some embodiments, the motion description text is a description of the motion trajectory or action of the target object, typically in natural language. Pixel coordinate displacement refers to the displacement of each pixel from the initial image to the next frame, usually represented by a displacement field. The displacement field is calculated by combining the motion description text and the features of the initial image to generate the displacement of each pixel.
[0056] Please see Figure 3 In some embodiments, step S102 may include, but is not limited to, steps S301 to S303: Step S301: Obtain motion parameters from the motion description text.
[0057] Step S302: The target velocity inference submodule of the target motion simulation model performs velocity prediction on the motion parameters and initial image features to obtain the pixel velocity data of each pixel in the initial image.
[0058] Step S303: The target displacement inference submodule of the target motion simulation model calculates the pixel coordinate displacement based on the pixel velocity data and the preset inter-frame time period.
[0059] In step S301 of some embodiments, natural language processing technology can be used to parse the motion description text to obtain motion parameters. The specific process is the same as the logic of the specific embodiment of step S202, and will not be described in detail here.
[0060] In step S302 of some embodiments, the target velocity inference submodule is used to infer the velocity of each pixel. Its implementation process is similar to that of embodiments related to analytical formula (1). Specifically: First, the initial image features are dimensionally transformed using a multilayer perceptron to extract intermediate features from the image. Then, the Hadamard product is calculated based on the intermediate image features and motion parameters to obtain pixel velocity data. The pixel velocity data is the velocity data of each pixel in the initial image.
[0061] In step S303 of some embodiments, the target displacement inference submodule is used to infer the displacement of an object or scene between different time steps (frames). The inter-frame time period refers to the time interval between adjacent frames in an image sequence, typically a fixed time step. Pixel coordinate displacement refers to the change in position of each pixel between time steps, including the magnitude and direction of each pixel's displacement from the current frame to the next frame. Its specific calculation process is similar to that of the embodiment of step S203, and will not be repeated here.
[0062] Steps S301 to S303, as illustrated in the embodiments of this application, combine motion description text with initial image features to accurately predict the motion velocity and coordinate displacement of each pixel, thereby generating simulation animations that better conform to physical laws and temporal consistency. This technical solution not only improves the accuracy of the image generation process but also optimizes the motion simulation model through physical modeling, solving the problems of physical inconsistency and inter-frame discontinuity in traditional image generation methods.
[0063] In step S103 of some embodiments, it should be noted that physical image features refer to physical feature maps in the next frame of the current frame, which represent object features in the scene of the next frame.
[0064] In this embodiment, the GridSample function can be used to perform feature map sampling on the initial image features and pixel coordinate displacements. The process can be referred to in the following analytical expression: (4), in, Represents physical image features. This represents the initial image features. This represents the pixel coordinate displacement.
[0065] Please see Figure 4 In some embodiments, step S104 may include, but is not limited to, steps S401 to S404: Step S401: Perform linear mapping on the physical image features to obtain the predicted key vector and the predicted value vector.
[0066] Step S402: Perform linear mapping on the initial image features to obtain the initial query vector.
[0067] Step S403: Calculate the weights based on the predicted key vector and the initial query vector to obtain the temporal attention weights.
[0068] Step S404: Perform a weighted summation based on the temporal attention weights and the predicted value vector to obtain the temporal attention features.
[0069] In step S401 of some embodiments, physical image features can be linearly mapped through multiple fully connected layers.
[0070] In step S402 of some embodiments, the initial image features can be linearly mapped through multiple fully connected layers.
[0071] In step S403 of some embodiments, the calculation process of the temporal attention weights can refer to the following analytical expression: (5), Where i represents the position index of the initial image, and j represents the position index of the legend image features. This represents the temporal attention weights. q i This represents the initial query vector. k j This represents the predicted key vector, where n represents the image sequence number. K represents the preset total number of frames. d represents q i and k j The vector dimension.
[0072] In step S404 of some embodiments, the calculation process of the temporal attention features can refer to the following analytical expression: (6), in, This represents temporal attention features. This represents the predicted value vector.
[0073] Steps S401 to S404 of this application embodiment propose an efficient simulation animation generation method based on physical and image features by introducing a temporal attention mechanism. First, by linearly mapping the physical image features, predicted key vectors and predicted value vectors are obtained, providing the necessary physical basis for subsequent temporal attention calculations. Next, by linearly mapping the initial image features, an initial query vector is obtained, which provides the foundation for calculating temporal dependencies. Then, based on the relationship between the predicted key vector and the initial query vector, temporal attention weights are calculated to ensure that different positions in the image can be appropriately weighted according to temporal changes. Finally, a weighted sum of the temporal attention weights and the predicted value vectors is performed to obtain the temporal attention features, thereby ensuring high consistency and physical plausibility of the generated simulation animation across multiple frames.
[0074] In step S105 of some embodiments, the candidate image is used as the image in the next frame after the initial image in the target simulation animation.
[0075] Please see Figure 5 In some embodiments, step S105 may also include, but is not limited to, steps S501 to S504: Step S501: Convolve the initial image features to obtain convolved image features.
[0076] Step S502: Activation processing is performed on the convolutional image features to obtain intermediate image features.
[0077] Step S503: The intermediate image features, physical image features and temporal attention features are fused to obtain fused image features.
[0078] Step S504: Construct an image from the fused image features to obtain candidate images.
[0079] In step S501 of some embodiments, a convolutional neural network (CNN) can be used to process the initial image, and the convolutional layer extracts feature information at different levels through multiple convolutional kernels.
[0080] In step S502 of some embodiments, the activation process is to perform a nonlinear transformation on the convolutional image features. The purpose is to introduce nonlinear factors so that the network can process complex image features and avoid the network from behaving as a linear function. Common activation functions include ReLU, Sigmoid, Tanh, etc.
[0081] In step S503 of some embodiments, the process of fusing inter-image features, physical image features, and temporal attention features can be referred to the following analytical expression: (7), in, This represents the features of the fused image. Represents physical image features. This represents temporal attention features. and These are all preset coefficients used to balance the contributions of physical image features and temporal attention features.
[0082] In step S504 of some embodiments, candidate images can be generated by decoding the fused image features through a pre-trained image generation model.
[0083] Steps S501 to S504, as illustrated in this embodiment, process the initial image features through convolution operations to obtain convolutional image features. Subsequently, activation processing is performed to further extract high-level features from the image, resulting in intermediate image features. Then, by fusing the intermediate image features, physical image features, and temporal attention features, it is ensured that the subsequently generated candidate images have high temporal consistency with the initial image and accurately follow physical laws.
[0084] It should be noted that after step S105, the obtained candidate image will be used as the next frame image of the initial image. If it is necessary to continue generating subsequent animation frames, return to step S101, and use the candidate image as the initial image for the new loop step, and execute steps S101 to S105 again to obtain a new candidate image, which will be the third frame image of the target simulation animation. Repeat this process until the number of frames reaches a preset value. It can be understood that in the same simulation animation generation task, the process of repeatedly executing the steps does not require repeatedly obtaining motion description text.
[0085] In step S106 of some embodiments, the initial image and all candidate images are arranged and spliced in chronological order to form a continuous animation sequence, which is the target simulation animation.
[0086] Please see Figure 6 This application also provides an animation generation apparatus that can implement the above-described animation generation method. The apparatus includes: The image feature extraction module 601 is used to acquire an initial image and extract image features from the initial image to obtain the initial image features; The displacement inference module 602 is used to acquire motion description text and perform coordinate displacement inference on motion description text and initial image features through a preset target motion simulation model to obtain the pixel coordinate displacement of each pixel in the initial image. The physical feature sampling module 603 is used to sample the feature map based on the initial image features and pixel coordinate displacement to obtain physical image features; The temporal attention module 604 is used to perform temporal attention calculation on physical image features and initial image features to obtain temporal attention features; The image generation module 605 is used to generate candidate images based on initial image features, physical image features, and temporal attention features. Animation building module 606 is used to construct an ordered sequence based on the initial image and candidate images to obtain the target simulation animation.
[0087] The specific implementation of this animation generation device is basically the same as the specific implementation of the animation generation method described above, and will not be repeated here.
[0088] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described animation generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0089] Please see Figure 7 , Figure 7The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 using the animation generation method of the embodiments of this application. The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.
[0090] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described animation generation method.
[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0092] The animation generation method, apparatus, electronic device, and storage medium provided in this application extract initial image features from an initial image. These features, along with motion description text, are then input into a target motion simulation model, which infers the pixel coordinate displacement of each pixel in the initial image. Next, feature map sampling is performed on the displacement and the initial image features to obtain physical image features, ensuring that subsequently generated images conform to physical laws. Then, temporal attention features are generated by performing temporal attention calculations on the initial image features and physical image features, effectively capturing dynamic changes and spatial dependencies between adjacent frames and ensuring high consistency between consecutive frames. Finally, the generated candidate images are integrated into an ordered sequence to construct the target simulation animation. This application not only improves the temporal consistency of the animation but also ensures that the displayed content conforms to physical laws, ultimately improving the accuracy of animation generation.
[0093] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0094] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0096] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or appropriate combinations thereof.
[0097] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0098] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0099] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0100] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0102] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0103] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. An animation generation method, characterized in that, The method includes: An initial image is acquired, and image features are extracted from the initial image to obtain the initial image features; Obtain motion description text, and use a preset target motion simulation model to infer coordinate displacement of the motion description text and the initial image features to obtain the pixel coordinate displacement of each pixel in the initial image. Based on the initial image features and the pixel coordinate displacement, feature map sampling is performed to obtain physical image features; Temporal attention features are obtained by performing temporal attention calculation on the physical image features and the initial image features. Image generation is performed based on the initial image features, the physical image features, and the temporal attention features to obtain candidate images; An ordered sequence is constructed based on the initial image and the candidate images to obtain the target simulation animation.
2. The method according to claim 1, characterized in that, Before performing coordinate displacement inference on the motion description text and the initial image features using a preset target motion simulation model, the method further includes pre-training the target motion simulation model, specifically including: Obtain the initial image of the sample, and extract image features from the initial image of the sample to obtain the sample image features; The sample description text is obtained, and the initial velocity inference submodule of the preset initial motion simulation model is used to perform velocity inference on the sample image features and the sample description text to obtain the predicted velocity data of each pixel in the sample starting image. The initial displacement inference submodule of the initial motion simulation model performs displacement calculation on the predicted velocity data to obtain the reference coordinate displacement of each pixel in the initial image of the sample. Based on the predicted velocity data, the velocity curl term and velocity divergence term are calculated, and based on the reference coordinate displacement, the collision penalty term is calculated. The initial motion simulation model is adjusted according to the velocity curl term, the velocity divergence term, and the calculated collision penalty term to obtain the target motion simulation model.
3. The method according to claim 2, characterized in that, The step of calculating the velocity curl and velocity divergence terms based on the predicted velocity data, and calculating the collision penalty term based on the reference coordinate displacement, includes: The curl of the predicted velocity data is calculated to obtain the velocity curl; The curl norm is obtained by calculating the Euclidean norm of the velocity curl. The predicted velocity data is subjected to divergence calculation to obtain velocity divergence; The Euclidean norm of the velocity divergence is calculated to obtain the divergence norm. The displacement of the reference coordinates and the preset collision position are used to calculate the collision displacement offset. The collision displacement offset is calculated using the Manhattan norm to obtain the collision displacement norm; The expected value of the curl norm is used as the velocity curl term, the expected value of the divergence norm is used as the velocity divergence term, and the expected value of the collision displacement norm is used as the collision penalty term.
4. The method according to claim 1, characterized in that, The step of inferring the coordinate displacement of each pixel in the initial image by performing coordinate displacement inference on the motion description text and the initial image features through a preset target motion simulation model includes: Obtain motion parameters from the motion description text; The target velocity inference submodule of the target motion simulation model performs velocity prediction on the motion parameters and the initial image features to obtain pixel velocity data for each pixel in the initial image. The target displacement inference submodule of the target motion simulation model calculates the pixel coordinate displacement by using the pixel velocity data and the preset inter-frame time period.
5. The method according to claim 4, characterized in that, The target velocity inference submodule of the target motion simulation model performs velocity prediction on the motion parameters and the initial image features to obtain pixel velocity data for each pixel in the initial image, including: The initial image features are subjected to dimensionality transformation to obtain intermediate image features; The Hadamard product is calculated based on the intermediate image features and the motion parameters to obtain the pixel velocity data.
6. The method according to any one of claims 1 to 5, characterized in that, The step of performing temporal attention calculation on the physical image features and the initial image features to obtain temporal attention features includes: Linear mapping is performed on the physical image features to obtain the predicted key vector and the predicted value vector; The initial image features are linearly mapped to obtain the initial query vector; The temporal attention weights are obtained by calculating the weights based on the predicted key vector and the initial query vector. The temporal attention features are obtained by weighted summation of the temporal attention weights and the predicted value vector.
7. The method according to any one of claims 1 to 5, characterized in that, The step of generating candidate images based on the initial image features, the physical image features, and the temporal attention features includes: The initial image features are convolved to obtain convolved image features; The convolutional image features are activated to obtain intermediate image features; The intermediate image features, the physical image features, and the temporal attention features are fused to obtain the fused image features; The fused image features are used to construct the candidate image.
8. An animation generation device, characterized in that, The device includes: An image feature extraction module is used to acquire an initial image and extract image features from the initial image to obtain initial image features; The displacement inference module is used to acquire motion description text and perform coordinate displacement inference on the motion description text and the initial image features through a preset target motion simulation model to obtain the pixel coordinate displacement of each pixel in the initial image. The physical feature sampling module is used to perform feature map sampling based on the initial image features and the pixel coordinate displacement to obtain physical image features; The temporal attention module is used to perform temporal attention calculations on the physical image features and the initial image features to obtain temporal attention features; The image generation module is used to generate candidate images based on the initial image features, the physical image features, and the temporal attention features. An animation construction module is used to construct an ordered sequence based on the initial image and the candidate images to obtain a target simulation animation.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.