Vector graphic animation generation method and system based on implicit neural representation and text-to-video diffusion model

Abstract

The invention discloses a vector graphic animation generation method and system based on implicit neural representation and a text-to-video diffusion model, and belongs to the field of computer graphics. The method comprises the following steps: generating an initial static video corresponding to a vector graph based on n layered implicit neural representation networks distributed side by side; using a text-to-video diffusion model to extract action information in the text cue word to generate a coarse-grained animation video corresponding to the initial static video; and refining the coarse-grained animation video to obtain a fine-grained animation video. According to the invention, the efficiency and quality of vector animation generation are improved, and the threshold of animation production is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a technology for automatically generating vector graphics animations. Specifically, the present invention relates to a method and system for generating vector graphics animations based on implicit neural representation and a text-to-video diffusion model. Background Art

[0002] Vector graphics are widely used in industrial design, animation, web design and other fields due to their scalability and user-friendliness. Compared with traditional raster images, vector graphics are composed of well-defined graphic elements, such as ellipses, lines and Bezier curves. This structure ensures that vector graphics will not lose clarity when scaled, and is suitable for a variety of display devices and resolutions. However, the process of vector animation production usually requires a lot of manual operations, including the establishment of complex skeletons, the setting of motion trajectories and time control, which leads to low efficiency and huge workload in animation production.

[0003] Some existing methods attempt to process vector graphics animations through computer automation, but most methods have problems such as insufficient flexibility, low animation quality, and weak constraints on color and shape. For example, some methods convert vector graphics into raster images before animation, resulting in the loss of the original properties of the graphics and the lack of consistency in shape and color of the generated animation. Therefore, a new automated method is needed to effectively generate high-quality vector animations to lower the production threshold and improve efficiency. Summary of the invention

[0004] In view of the above problems, the present invention discloses a vector graphics animation generation method based on implicit neural representation and text-to-video diffusion model, which improves the efficiency and quality of vector animation generation and lowers the threshold of animation production.

[0005] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0006] A vector graphics animation generation method based on implicit neural representation and text-to-video diffusion model, the method comprising:

[0007] Based on n hierarchical implicit neural representation networks distributed side by side, an initial static video corresponding to the vector graphics is generated;

[0008] Extracting action information from text prompt words using a text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video;

[0009] The coarse-grained animation video is refined to obtain a fine-grained animation video.

[0010] Furthermore, the generating of the initial static video corresponding to the vector graphics based on the n hierarchical implicit neural representation networks distributed side by side includes:

[0011] Parse the vector graphics to get the layered color information c l and each layer has a shape area with a corresponding color l∈[1,L], L is the total number of layers of the hierarchical implicit neural representation network;

[0012] The hierarchical implicit neural representation network is optimized using the mean square error loss of each layer so that the color opacity m of each layer output l A shape area with the color corresponding to the input vector graphic Consistency;

[0013] Get the background color c0 and fill the canvas with the background color c0;

[0014] In each layer, based on the color opacity m l , the layered color information c l and the shape area Perform color filling to generate the i-th initial video frame V in the initial static video i .

[0015] Furthermore, the extracting action information in the text prompt words by using the text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video includes:

[0016] In video frame Add noise ∈ to get the noisy video frame V after the tth noise addition i ' (t) ; where t is a natural number, is the i-th video frame in the initial static video;

[0017] The text prompt word and the noise-added video frame V i ' (t) Input into the text-to-video diffusion model to get the predicted noise Among them, c represents the text prompt word, Represents a text-to-video diffusion model;

[0018] Based on the prediction noise The difference with the noise ∈ Optimize the i-th hierarchical implicit neural representation network so that the i-th hierarchical implicit neural representation network outputs a video frame

[0019] Let t = t + 1, and re-execute the above process in the video frame Add noise ∈ to get the noisy video frame V after the tth noise addition i ' (t) , until the i-th layered implicit neural representation network converges, thus obtaining the i-th video frame in the coarse-grained animation video

[0020] Furthermore, the text-to-video diffusion model includes: a noise prediction model based on a transformer architecture.

[0021] Furthermore, the step of refining the coarse-grained animation video to obtain a fine-grained animation video includes:

[0022] Compute the vector graphics into video frames The optical flow of the video frame is the i-th video frame in the coarse-grained animation video;

[0023] Based on the optical flow calculation result, the parameters of the vector graphics are displaced to obtain the i-th video frame in the fine-grained animation video.

[0024] A vector graphics animation generation system based on implicit neural representation and text-to-video diffusion model, the system comprising:

[0025] A static video generation module, used for generating an initial static video corresponding to a vector graphic based on n hierarchical implicit neural representation networks distributed side by side;

[0026] A coarse-grained video generation module, used to extract action information in text prompt words by using a text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video;

[0027] The fine-grained video generation module is used to refine the coarse-grained animation video to obtain a fine-grained animation video.

[0028] An electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the vector graphics animation generation method based on implicit neural representation and text-to-video diffusion model described in any one of the above items is implemented.

[0029] A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement any of the above-mentioned vector graphics animation generation methods based on implicit neural representation and text-to-video diffusion model.

[0030] Compared with the existing technology, the present invention significantly improves the efficiency and quality of vector animation generation, can handle complex graphics and motion expressions, and maintain the consistency of color and shape, thereby lowering the threshold for animation production. It can be widely used in animation production, game development, advertising design and other fields. Users only need to provide basic vector graphics and text prompts to automatically generate high-quality animation effects, providing convenience for creation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowchart of the method for vector graphics animation generation based on implicit neural representation and text-to-video diffusion model. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0033] The purpose of the present invention is to provide a vector graphics animation generation method based on implicit neural representation and text-to-video diffusion model, which reconstructs the input vector graphics through hierarchical implicit neural representation and automatically generates animation using text prompt information. Figure 1 shown.

[0034] Step 1: Generate an initial static video corresponding to the vector graphics based on n side-by-side distributed hierarchical implicit neural representation networks.

[0035] The content of this step can be divided into two parts: vector graphics reconstruction and static video generation.

[0036] In vector graphics reconstruction, the vector graphics file input by the user is received, and the color information of each layer and the shape area with corresponding color in each layer are analyzed. Assume that there are L layers, and the layered color information is recorded as (c1, c2, ..., c L ), and then use the layered implicit neural representation to reconstruct the image. The traditional implicit neural representation of an image takes coordinates (X, Y) as input and the corresponding colors (R, G, B) as output. In order to adapt to the layered characteristics of vector graphics files, the layered neural implicit representation outputs the color opacity of each layer (m1, m2, ..., m L ), the image is the result of coloring layer by layer according to the color opacity. Specifically, when rendering an image, the canvas is first filled with the background color c0, and then the opacity m of the color at each coordinate is calculated through the neural network for each layer. l , according to the opacity m l Overlay the corresponding color c on the image l The neural network with hierarchical implicit neural representation is optimized using mean squared error loss at each layer so that the graphics it represents can reconstruct the input vector graphics.

[0037] In static video generation, the present invention directly copies the reconstructed hierarchical implicit neural representation network multiple times to adapt to the number of video frames to generate an initial static video.

[0038] Step 2: Use the text-to-video diffusion model to extract the action information in the text prompt words to generate a coarse-grained animation video corresponding to the initial static video.

[0039] The present invention inputs the initial video frame into a pre-trained fixed-parameter text-to-video diffusion model to perform multiple rounds of repeated optimization on the parameters of the hierarchical implicit neural representation neural network. The text-to-video diffusion model used is a noise prediction model based on the transformer architecture that has been pre-trained in a natural video dataset and does not require further fine-tuning. Let the initial video frame be V. During the optimization process, noise ∈ is added to obtain a noise video V′=α t V+σ t ∈, where t is the number of steps of adding noise, α t and σ t is a parameter related to t, using the text-to-video diffusion model Get the predicted noise Where c is the text prompt word, and the fractional distillation sampling loss is calculated as the difference between the added noise and the predicted noise Among them, w(t) is a parameter related to t, and back propagation is performed to optimize all hierarchical implicit neural expression networks. After multiple rounds of optimization, the initial static video will gradually generate motion, and the difference between the added noise and the predicted noise will gradually decrease, thereby achieving the extraction of motion information from the diffusion model.

[0040] In one embodiment, the process of generating a coarse-grained animation video includes the following steps.

[0041] Step 2.1: In the video frame Add noise ∈ to get the noisy video frame V after the tth noise addition i ' (t) ; where t is a natural number, is the i-th video frame in the initial static video;

[0042] Step 2.2: Combine the text prompt word and the noisy video frame V i ' (t) Input into the text-to-video diffusion model to get the predicted noise Among them, c represents the text prompt word, Represents a text-to-video diffusion model;

[0043] Step 2.3: Based on prediction noise The difference with the noise ∈ Optimize the i-th hierarchical implicit neural representation network so that the i-th hierarchical implicit neural representation network outputs a video frame

[0044] Step 2.4: Let t = t + 1, and re-execute the above steps in the video frame Add noise ∈ to get the noisy video frame V after the tth noise addition ′(t) , until the i-th layered implicit neural representation network converges, thus obtaining the i-th video frame in the coarse-grained animation video

[0045] Step 3: Refine the coarse-grained animation video to obtain a fine-grained animation video.

[0046] The present invention uses the optical flow method to calculate the optical flow from the original graphics to each frame of animation, and displaces the original vector graphics parameter points according to the optical flow to ensure the smooth transition of the animation. When necessary, the mean square error loss is used to directly optimize each frame to enhance flexibility.

[0047] In summary, since directly optimizing the parameters of vector graphics using the text-to-video diffusion model will generate a huge data domain gap in the middle, the implicit neural representation has many good properties, that is, the implicit neural representation not only has the characteristics of infinite resolution like vector graphics, but also provides shape operability, which can produce relatively free movement under the conditions of constraining shape and color. Therefore, the present invention innovatively makes the implicit neural representation serve as a bridge between the vector graphics parameter domain and the diffusion model, thereby generating a higher quality coarse-grained animation video.

[0048] Based on the same concept, the present invention also discloses a vector graphics animation generation system based on implicit neural representation and text-to-video diffusion model, the system comprising:

[0049] A static video generation module, used for generating an initial static video corresponding to a vector graphic based on n hierarchical implicit neural representation networks distributed side by side;

[0050] A coarse-grained video generation module, used to extract action information in text prompt words by using a text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video;

[0051] The fine-grained video generation module is used to refine the coarse-grained animation video to obtain a fine-grained animation video.

[0052] Based on the same concept, the present invention also discloses an electronic device, characterized in that the electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned vector graphics animation generation methods based on implicit neural representation and text-to-video diffusion model.

[0053] Based on the same concept, the present invention also discloses a computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the above-mentioned vector graphics animation generation methods based on implicit neural representation and text-to-video diffusion model is implemented.

[0054] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for generating vector graphics animation based on implicit neural representation and text-to-video diffusion model, characterized in that: The method comprises: Based on n hierarchical implicit neural representation networks distributed side by side, an initial static video corresponding to the vector graphics is generated; Extracting action information from text prompt words using a text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video; The coarse-grained animation video is refined to obtain a fine-grained animation video.

2. The method according to claim 1, characterized in that: The method of generating an initial static video corresponding to the vector graphics based on n hierarchical implicit neural representation networks distributed side by side includes: Parse the vector graphics to get the layered color information c l and each layer has a shape area with a corresponding color L is the total number of layers in the hierarchical implicit neural representation network; The hierarchical implicit neural representation network is optimized using the mean square error loss of each layer so that the color opacity m of each layer output l A shape area with the color corresponding to the input vector graphic Consistency; Get the background color c0 and fill the canvas with the background color c0; In each layer, based on the color opacity m l , the layered color information c l and the shape area Perform color filling to generate the i-th initial video frame V in the initial static video i .

3. The method according to claim 1, characterized in that The method of extracting action information from text prompt words by using a text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video includes: In the video frame V i t Add noise ∈ to get the noisy video frame V after the tth noise addition i ' (t) ; where t is a natural number, V i t =0 is the i-th video frame in the initial static video; The text prompt word and the noise added video frame V i ' (t) Input into the text-to-video diffusion model to get the predicted noise Among them, c represents the text prompt word, Represents a text-to-video diffusion model; Based on the prediction noise The difference with the noise ∈ Optimize the i-th hierarchical implicit neural representation network so that the i-th hierarchical implicit neural representation network outputs a video frame V i t+1 ; Let t = t + 1, and re-execute the above process in video frame V i t Add noise ∈ to get the noisy video frame V after the tth noise addition i ' (t) , until the i-th layered implicit neural representation network converges, thus obtaining the i-th video frame V in the coarse-grained animation video i c .

4. The method according to claim 1, characterized in that The text-to-video diffusion model includes: a noise prediction model based on a transformer architecture.

5. The method according to claim 1, characterized in that The step of refining the coarse-grained animation video to obtain a fine-grained animation video includes: Calculate the vector graphics into video frame V i c The optical flow of the video frame V i c is the i-th video frame in the coarse-grained animation video; Based on the optical flow calculation result, the parameters of the vector graphics are displaced to obtain the i-th video frame in the fine-grained animation video.

6. A vector graphics animation generation system based on implicit neural representation and text-to-video diffusion model, characterized in that: The system comprises: A static video generation module, used for generating an initial static video corresponding to a vector graphic based on n hierarchical implicit neural representation networks distributed side by side; A coarse-grained video generation module, used to extract action information in text prompt words by using a text-to-video diffusion model to generate a coarse-grained animation video corresponding to the initial static video; The fine-grained video generation module is used to refine the coarse-grained animation video to obtain a fine-grained animation video.

7. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the vector graphics animation generation method based on implicit neural representation and text-to-video diffusion model as described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the vector graphics animation generation method based on implicit neural representation and text-to-video diffusion model as described in any one of claims 1-5 is implemented.

Images