Image vectorization method and device based on neural diffusion curve

Abstract

The invention discloses an image vectorization method and device based on a neural diffusion curve. According to the method, geometric control point parameters of multiple sections of Bezier curves are predicted from end to end in an input grating image through a deep neural network, and the main contour and color structure of the image are accurately represented in a Bezier curve form; color control points on the two sides of the curve are automatically extracted through a micro color sampling module, and physically consistent diffusion curve representation is constructed; and performing neural approximation on the resolving process of the diffusion equation by adopting a rendering network based on a Fourier neural operator to realize end-to-end micro-reconstruction of the image generated by the prediction curve, thereby reversely optimizing the geometric prediction of the curve by using a reconstruction error. According to the method, manual intervention is not needed, image vector representation which is high in fidelity, infinite in scaling, lossless in editing and capable of supporting natural layering can be automatically generated while physical rendering consistency is kept, precision and efficiency are improved, and the method is suitable for the fields of image vectorization, digital art creation, micro rendering, graphic design and the like.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and image vectorization, and specifically to an image vectorization method and apparatus based on neural diffusion curves. Background Technology

[0002] Vector images, due to their advantages such as resolution independence, small file size, and high editability, have been widely used in fields such as digital art, industrial design, publishing, and computer graphics. The core task of image vectorization technology is to convert raster images into editable vector representations composed of geometric curves and color regions.

[0003] Early image vectorization techniques were mainly based on edge detection, region segmentation, and curve fitting, which achieved good results when processing simple images with clear outlines and well-defined color partitions. However, when faced with continuous gradients, soft shadows, complex textures, or lighting variations common in natural images, these methods often struggle to simultaneously ensure the accuracy of geometric structure and the smoothness of color distribution. This can easily lead to problems such as boundary breaks, color blocking, geometric distortion, or overfitting. Furthermore, they typically require significant manual intervention and post-processing corrections, resulting in low automation and processing efficiency.

[0004] To better represent smooth color transitions, researchers have proposed diffusion curves, a classic vector representation. By specifying colors on both sides of the curve and solving the Poisson diffusion equation, continuous and natural gradations and shadows can be reconstructed physically and consistently, significantly improving the visual fidelity of vector images. However, traditional diffusion curve methods rely on numerical solvers such as finite difference or finite element methods for discretization rendering. This process is non-differentiable, meaning that the final reconstruction error cannot be backpropagated to the curve's geometric and color control points. This makes it difficult to seamlessly integrate with deep learning frameworks. Therefore, the curves still need to be drawn manually or generated using heuristic algorithms, making it difficult to achieve fully automated, high-precision vectorization.

[0005] In recent years, deep learning has demonstrated powerful capabilities in edge detection, contour extraction, and geometric regression tasks, giving rise to a variety of automated vectorization schemes based on neural networks. While these methods can predict vector contours or piecewise Bézier curves relatively well, most remain at the geometric level and lack physical modeling of the color diffusion process. At the same time, due to the lack of a fully differentiable closed loop from "curve prediction - diffusion rendering - image reconstruction," the network cannot directly optimize curve parameters using the final image reconstruction error, making it difficult to uniformly learn the intrinsic correspondence between geometric structure and color gradation.

[0006] To address the aforementioned issues, this invention proposes an image vectorization method based on neural diffusion curves. This method deeply integrates deep neural networks with diffusion curve theory. Through end-to-end differentiable curve prediction, differentiable color sampling, and a fast rendering module based on Fourier neural operators, it achieves highly unified modeling of geometric structures and color gradients. This generates high-fidelity, editable, and physically consistent vector images without human intervention, significantly improving accuracy, robustness, and practicality. Summary of the Invention

[0007] This invention relates to an image vectorization method and apparatus based on neural diffusion curves. The purpose of this invention is to address the problems in existing technologies, such as the difficulty in uniformly modeling curve geometry and color gradients, and the inability to simultaneously optimize geometry and color within the same framework due to the non-differentiability of the diffusion process. This allows for a holistic vectorization process that can be used for end-to-end training.

[0008] The objective of this invention is achieved through the following technical solution: an image vectorization method based on neural diffusion curves, the method comprising the following steps:

[0009] Step 1: Acquire the input raster image;

[0010] Step 2: Construct a curve extraction network based on an encoder-decoder architecture to predict several feature curves in the raster image and represent them in piecewise cubic Bézier form;

[0011] Step 3: Based on the predicted curve, perform differentiable color sampling on both sides of the original image to construct a diffusion curve containing geometric control points and color control points;

[0012] Step 4: Calculate the color gradient field based on the diffusion curve, and use the Boolean mask that identifies the fixed boundary points and the color components of the boundary points as input to the curve rendering network.

[0013] Step 5: Construct a curve rendering network based on Fourier neural operators, perform neural approximation solution on the Poisson equation driven by the diffusion curve, and generate the reconstructed image.

[0014] Step six, adopt a two-stage training strategy: the first stage supervises the training of the curve extraction network and the curve rendering network; the second stage performs end-to-end joint optimization of the curve extraction network based on the reconstruction loss.

[0015] Step 7: Obtain the final vector image based on the neural diffusion curve using the trained curve extraction network and curve rendering network.

[0016] Furthermore, the curve extraction neural network includes a convolutional encoder and a multi-layer attention-based decoding structure; wherein, the convolutional encoder is used to process the input image. Generate multi-scale feature maps and fuse them to form sequence features. The decoding structure uses learnable query vectors as an implicit representation of the curve:

[0017]

[0018] in, For the initial query vector set, For the first Query vectors, with dimensions of _ , Let be the number of query vectors. The query vectors are updated at each layer using self-attention and cross-attention mechanisms, satisfying:

[0019]

[0020] in Indicates the first The layer decoder's update operator, based on the updated query vector, outputs the existence probability of each curve in the curve prediction head. Number of segments and control point coordinates ,in Indicates the first The first curve Coordinates of control points , This represents the maximum number of segments the curve can be divided into.

[0021] Furthermore, each predicted curve described in step two is represented by multiple cubic Bézier curves, where each curve contains the number of segments predicted by the network. And splice the segments inside the curve; each segment of the curve consists of four control points. The determined geometry satisfies the equation of a cubic Bézier curve:

[0022]

[0023] in, These are curve parameters. For two adjacent segments within the same curve... and The connection points satisfy the G1 continuity constraint, that is:

[0024]

[0025]

[0026] The geometric structure of a single complete diffusion curve is formed by continuously splicing together multiple Bezier segments.

[0027] Furthermore, by parametrically sampling the piecewise Bézier curve obtained in step two, a set of sampling points is obtained. and its corresponding curve parameters Where N is the number of sampling points; simultaneously, a tangent vector is calculated for each sampling point:

[0028]

[0029] And denoted at discrete sampling points Rotate the tangent vector counterclockwise around the origin. Obtain the normal vector:

[0030]

[0031] Using the normal vector to offset left and right distances within the image domain Obtain the left and right sampling points, where Indicates the fuzzy radius of the curve. Indicates the preset scaling factor:

[0032]

[0033] The left and right colors are obtained at the above positions using differentiable bilinear interpolation:

[0034]

[0035] and color sequence and Differentiable aggregation yields a fixed-dimensional sequence of left and right color control points. and and the curve control points predicted in step two. The parameter structure that together constitutes the diffusion curve .

[0036] Furthermore, in step four, the multiple diffusion curves obtained in step three are resampled to obtain a series of curve sampling points. Simultaneously, based on the left and right color control point sequences obtained in step three, in each resampling parameter The left and right color values ​​can be obtained by differentiable interpolation. and The corresponding unit normal vector is calculated. Construct a color gradient field at the curve position:

[0037]

[0038] And zeros are applied to pixels outside the curve, so that the color gradient only jumps near the curve;

[0039] The projection of the curve onto the discrete pixel grid is marked as a Dirichlet boundary, forming a Boolean mask:

[0040]

[0041] in This represents a pixel position in the discrete pixel domain, where its coordinates are pairs of two-dimensional integers. ,when At that time, the pixel is located on the projected boundary of the diffusion curve; the boundary color component is assigned at the position satisfying the Dirichlet boundary. ;

[0042] Finally, the gradient field Boolean mask and boundary color Composition forms a tensor:

[0043]

[0044] Tensor X is used as the input to the curve rendering network.

[0045] Furthermore, in step five, the curve rendering network first processes the input tensor... Concatenated with normalized grid coordinates, and mapped to an initial feature field of fixed width via a channel blending layer. Subsequently, the initial feature field passes through multiple layers of Fourier operator modules, and in each layer, it is updated by combining local spatial domain mapping with global frequency domain operations. The update form is expressed as:

[0046]

[0047] in It is a channel-wise nonlinear activation function. It is a local linear mapping. and These are the two-dimensional Fast Fourier Transform and its inverse transform, respectively. A learnable frequency domain kernel defined on the truncated frequency domain; This represents the feature field after the (t+1)th update;

[0048] pass Layer-by-layer Fourier block iterations, using a rendering network based on the Fourier Neural Operator (FNO) to approximate the input tensor in a data-driven manner. To the solution of the diffusion equation The mapping operator in the final feature field The predicted image is generated by linear projection. .

[0049] Furthermore, in step six, the supervised training of the curve extraction network and the curve rendering network in the first stage of the two-stage training is based on the matching relationship between the predicted curve and the real curve. Hungarian matching is used to establish a one-to-one correspondence between the predicted curve and the real curve in each input sample. For any real curve index... With arbitrary prediction curve index The matching cost is defined as:

[0050]

[0051] in For the first The existence probability of the prediction curve and For balance coefficient, This is the bidirectional Chamfer distance. and They represent the first The prediction curve and the first The set of discrete sampling points for the true curve. After obtaining the cost matrix, the optimal match is obtained by solving an optimization problem that minimizes the cost:

[0052]

[0053] in, Indicates all lengths of The set of permutations and combinations, for each true curve index , This represents the index of the predicted curve that matches it. The optimal permutation is obtained through... Establish a set of matching pairs The loss function after matching can then be supervised element by element.

[0054] Furthermore, in step six, the curve extraction network in the first stage of the two-stage training undergoes supervised training based on the annotation of the real diffusion curve, and its loss function includes the following five items:

[0055]

[0056] Among them, the items preceding each loss item Indicates the weight of the corresponding item;

[0057] Existential loss Using the binary classification cross-entropy form:

[0058]

[0059] in This represents the total number of queries. Indicates the first A query in real labels The corresponding predicted probabilities and class weights satisfy the following conditions: , Weights for background classes;

[0060] coordinate loss Using L1 norm form:

[0061]

[0062] in and These are the predicted and actual multi-segment Bezier control point sequences, respectively. In the implementation, different weights are applied to the endpoint control points and intermediate control points to strengthen the constraint on the endpoint positions.

[0063] Curve shape loss Chamfer distance based on multi-segment Bezier sampling is used:

[0064]

[0065] in and To utilize the cubic Bessel formula in step two The point set obtained by segmenting and sampling multiple control points;

[0066] For multi-segment curves, an additional segment prediction loss is introduced. It adopts a multi-class cross-entropy form:

[0067]

[0068] in To predict the actual number of segments The probability of this; simultaneously, to ensure the G1 continuity of multiple Bézier curves at the junctions between segments, a consistency loss based on the tangent direction is introduced. :

[0069]

[0070] in The total number of connection points between segments in all matching pairs. and The first The first prediction curve The tangent direction vectors of the preceding and following segments at the connection points between segments.

[0071] Furthermore, in step six, the curve rendering network in the first stage of the two-stage training is independently pre-trained based on real color gradient field data to minimize the following composite loss:

[0072]

[0073] in, This represents the weighting coefficient for each loss term; This provides a high-precision numerical solution corresponding to the same diffusion curve parameters. This represents the sum of squares over all pixels and color channels. This is a discrete Laplacian approximation using finite difference; the first term guarantees overall pixel-level reconstruction accuracy, and the second term explicitly constrains the network output to satisfy Dirichlet boundary conditions at fixed boundaries. The third point encourages predicted solutions to be spatially smooth and conform to the physical characteristics of diffusion processes.

[0074] Furthermore, in step six, in the second stage of the two-stage training strategy, the curve prediction network and the curve rendering network are connected end to end to form a closed-loop differentiable structure of "image-curve-image" from the input image to the predicted curve and then to the reconstructed image.

[0075] During this stage, the curve rendering network remains frozen and is used only as a differentiable operator. The end-to-end optimization objective is constructed based on the pixel-level consistency between the reconstructed image and the input image, and the optimization loss is in the form of:

[0076]

[0077] in For the input image, This is the reconstruction result obtained by freezing the rendering network.

[0078] On the other hand, the present invention also provides an image vectorization device based on neural diffusion curves, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the image vectorization method based on neural diffusion curves.

[0079] The main advantages of this invention include:

[0080] 1. Achieve high-precision geometric structure representation: Employ piecewise cubic Bézier curve representation and multi-layer Transformer decoding structure, enabling the model to generate target curves with strong continuity, smooth curvature, and fine control in image space, achieving higher geometric accuracy and expressive power compared to traditional segmentation or edge detection methods.

[0081] 2. Achieving highly consistent image reconstruction results: Through an end-to-end framework combining differentiable color sampling and Fourier neural operator (FNO) rendering network, this invention can simultaneously maintain the geometric accuracy and color consistency of curves, achieving image reconstruction results with natural texture transitions, clear boundaries, and high fidelity.

[0082] 3. Possesses strong scalability and practical application value: The model adopts a unified end-to-end learning framework, the curve prediction and rendering process is fully trainable, it can adapt to multi-resolution input and support vector editability, significantly improving its practicality and interactive efficiency in scenarios such as illustration generation, graphic design, engineering annotation and intelligent image vectorization. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a flowchart of an image vectorization method based on neural diffusion curves, as described in one embodiment of the present invention.

[0085] Figure 2 This is a specific flow of one embodiment of an image vectorization method based on neural diffusion curves according to one embodiment of the present invention.

[0086] Figure 3 This is a diagram of the curve extraction network architecture of an image vectorization method based on neural diffusion curves in one embodiment of the present invention.

[0087] Figure 4 This is a schematic diagram of color sampling for an image vectorization method based on neural diffusion curves, as shown in one embodiment of the present invention.

[0088] Figure 5 This is a schematic diagram of a curve rendering network input for an image vectorization method based on neural diffusion curves, as described in one embodiment of the present invention.

[0089] Figure 6 This is a curve rendering network architecture diagram of an image vectorization method based on neural diffusion curves in one embodiment of the present invention.

[0090] Figure 7 This is a schematic diagram of the vectorization result of an image vectorization method based on neural diffusion curves in one embodiment of the present invention.

[0091] Figure 8 This is a structural diagram of an image vectorization device based on neural diffusion curves in one embodiment of the present invention. Detailed Implementation

[0092] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0093] It should be noted that in the claims and description of this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] All documents mentioned in this invention are incorporated herein by reference as if each document were individually incorporated by reference. Furthermore, it should be understood that after reading the foregoing teachings of this invention, those skilled in the art can make various alterations or modifications to this invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0095] The method described in this invention has broad application potential in areas such as image vectorization, digital illustration, differentiable rendering, art creation assistance, and structured graphics extraction in complex scenes. Compared to traditional diffusion curve systems that rely on manual drawing or static algorithms, this invention can simultaneously generate high-quality geometric structures and physically consistent color gradients within an automated framework, making vector content more suitable for subsequent editing, animation production, style transfer, or fine reconstruction tasks. Taking the automatic generation of multi-segment diffusion curves from an input image as an example, this invention introduces the implementation process of a vectorization method based on neural networks and differentiable diffusion solutions. Specifically, it includes: preprocessing the input image to enhance geometric features; using an encoder-decoder curve extraction network to predict control points of multiple piecewise cubic Bézier curves; performing differentiable color sampling on both sides of the image based on the predicted geometry to obtain left and right color control points; constructing a color gradient field, boundary mask, and boundary color based on the diffusion curve; obtaining the reconstructed image by performing a neural approximation solution to the Poisson-type diffusion equation using Fourier neural operators; and achieving collaborative optimization of the geometric prediction network and the rendering network based on a two-stage training strategy.

[0096] Figure 1This is a flowchart of an image vectorization method based on neural diffusion curves according to an embodiment of the present invention. The method includes the following steps: Step 101, acquiring an input raster image; Step 102, constructing a curve extraction neural network to predict several feature curves in the image and representing them in piecewise cubic Bézier form; Step 103, constructing a diffusion curve containing geometric control points and color control points based on the differentiable colors on both sides of the original image according to the predicted curves; Step 104, calculating the color gradient field, Boolean mask, and boundary point colors based on the diffusion curves as input to the rendering network; Step 105, constructing a curve rendering network to perform neural approximation solving of the Poisson equation driven by the diffusion curves, generating a reconstructed image; Step 106, training using a two-stage training strategy; Step 107, outputting the final vector image based on the neural diffusion curves.

[0097] Specifically, this invention provides an image vectorization method based on neural diffusion curves. Figure 2 This document illustrates a specific implementation process of an image vectorization method based on neural diffusion curves, as described in this embodiment. The embodiment includes the following steps:

[0098] Step 1: Obtain the input image. The input image... The resolution is 256×256, and a single-channel edge map is generated using the Canny operator. Then, the original image and the edge map are concatenated along the channel dimension to construct a four-channel input tensor:

[0099]

[0100] This tensor serves as the input to the subsequent curve extraction network, enabling the network to utilize both color region information and explicit edge structures during the feature encoding stage, thereby enhancing its ability to perceive the geometric contours of curves.

[0101] Step 2: Construct a curve extraction neural network based on an encoder-decoder ( Figure 3 The algorithm predicts several feature curves in the image and represents them in piecewise cubic Bézier form.

[0102] The curve extraction neural network architecture used in this invention is a novel end-to-end network composed of a ResNet-FPN encoder and a multi-layer improved Transformer decoder. It dynamically represents each query as the control point of a Bézier curve and achieves spatially perceptual feature aggregation by sampling multiple reference points along the curve. Under the premise of ensuring end-to-end differentiability, it realizes fine modeling and layer-by-layer refinement of curve geometry.

[0103] In the encoding stage of the network, a pre-trained ResNet50 is used as the backbone network, based on the input image. Extract multi-scale features and obtain the feature set through the output of the intermediate layer. Subsequently, a Feature Pyramid Network (FPN) is introduced to fuse multi-scale features from top to bottom, unifying the output channels of each layer to 256 dimensions and forming a pyramid-shaped hierarchical representation. The highest resolution feature is then selected. Layers are used as the main working feature and are flattened into a sequence. This involves superimposing learnable 2D location embeddings to preserve image spatial structure information. The output sequence features at this stage... It possesses both edge sensitivity and high-level semantic awareness, providing rich geometric priors for subsequent curve modeling.

[0104] During the decoding phase of the network, the entire decoding process is implicitly represented by a fixed number of query vectors as the curve, with each query vector... This corresponds to a potential curve, which is recursively updated across layers. Let the sequence features output by the encoder be:

[0105]

[0106] No. The update of the layer decoder can be summarized as follows:

[0107]

[0108] in Indicates the first The update operator of the layer decoder, This is the updated query matrix.

[0109] The entire decoder consists of a multi-layered stacked structure, with each decoding layer containing three important modules: a self-attention module, a sequence feature-based cross-attention module, and a control point regression module. The self-attention module models the global spatial relationships between different curves, enabling the query vector to capture the layout relationships of other curves in the scene; let the input query matrix be... Then self-attention update can be written as:

[0110]

[0111] in The pre-normalized multi-head attention operator has the following basic form:

[0112]

[0113] in , , For the corresponding attention header query, key and value, The feature dimensions for each head. This step enables different curve candidates to share context and establish mutual arrangement awareness.

[0114] Subsequently, the updated query vector enters the cross-attention module. This module is used to combine the query vector with encoder sequence features. Interacting with each curve allows it to extract relevant information about its geometric location and semantics from image features. The cross-attention form is as follows:

[0115]

[0116] Among them Pre-normalized standard multi-head attention. This process enables the decoder to utilize spatial information related to the curve region in the encoded features, achieving coarse-to-fine geometric enhancement and feature aggregation.

[0117] After completing the attention interaction, the network proceeds to the Feed Forward Network (FFN):

[0118]

[0119] Through the above three steps, the query vector is gradually refined at each layer, so that its geometric meaning gradually converges to the feature pattern of the true diffusion curve.

[0120] Based on this, the present invention includes a control point regression module at each layer of the decoder, used to regress from the updated query vector. The prediction of the intermediate states of the curve includes: the probability of the curve's existence. Segmentation prediction Piecewise cubic Bézier curve control point matrix The control point regression process is represented as:

[0121]

[0122] in This represents a multi-head prediction network consisting of multiple layers of normalization and linear mapping. The control point outputs are normalized using the Sigmoid function to ensure that the coordinates are located at... Within the image domain, the decoder provides intermediate supervision layer by layer, which helps improve the robustness of the curve and the convergence of control points.

[0123] The final decoder outputs the probability of existence of each candidate curve. Number of segments and control point matrix ,in , The maximum number of segments in the curve is given. The final curve is pieced together in a piecewise cubic Bézier configuration to form a continuous diffusion curve geometry, providing a precise and differentiable curve parameter representation for subsequent color sampling and curve rendering networks.

[0124] Step 3: Based on the diffusion curve control point matrix obtained in Step 2, perform differentiable color sampling on each curve in the original image to obtain the color control point sequence on the left and right sides of the curve. Figure 4 Specifically, the control point sequence obtained in step two is first expanded using piecewise cubic Bézier curves. For the first... The segment curve, whose parametric form is:

[0125]

[0126] in These are the four control points for this section. These are the curve parameters. A global set of sampling points is obtained by sampling each curve segment at equal intervals and then concatenating the segments. and its corresponding curve parameters Simultaneously, a tangent vector is calculated for each sampling point.

[0127]

[0128] And denoted at discrete sampling points Rotate the tangent vector counterclockwise around the origin. Obtain the normal vector:

[0129]

[0130] The normal direction is stabilized through a sign consistency constraint, thereby ensuring that the direction determination on the left and right sides of the curve does not flip.

[0131] After obtaining the geometric sampling points and their normal directions, the normal vectors are offset to both sides in the image domain by a distance. (Based on the fuzzy radius of the curve) With preset ratio coefficient Confirmed, among which , ), to obtain the sampling positions on the left and right sides:

[0132]

[0133] Map the above sampling points to In the normalized coordinate domain, color values ​​are obtained from the input image through differentiable bilinear interpolation:

[0134]

[0135] In order to convert the discrete color sampling along the curve into a standardized vector representation, this step further aggregates the left and right color sequences into a fixed number of color control points while maintaining end-to-end differentiability. This ensures that the left and right color control points of each diffusion curve have a consistent dimension and can generate gradients synchronously as the geometric control points move.

[0136] The final obtained left and right color control points, together with the predicted geometric control points in step two, constitute a complete diffusion curve parameter representation, so that each curve has a differentiable, unified and structured description in terms of spatial geometry and color distribution, and directly serves as the input basis for the color gradient field construction and curve rendering network in step four.

[0137] Step four: Based on the diffusion curve obtained in step three, calculate the color gradient field, Boolean mask, and boundary color, which are used as inputs to the curve rendering network. Figure 5 First, after obtaining the piecewise Bézier representation of the curve, it is expanded into a continuous series of curve points using a uniform sampling strategy. Then, interpolation is performed on the left and right color control points using the normal direction, thereby generating a continuous left-side color source on both sides of the curve. With the color source on the right In the direction normal to the curve, at a fixed distance By appropriately pushing the left and right color sources outward from the main curve, they are kept separate in the grid pixel space, thus avoiding the problem of color source overlap caused by the small distance between adjacent pixels in the discrete grid.

[0138] Based on this, and according to the color gradient constraint proposed by the diffusion curve model, this step directly constructs a color gradient field on the curve. It takes a zero value at non-curved pixels, and only at curved positions does it express the magnitude of the color jump in the normal direction. Let the normal of the curve at a certain sampling point be... The color gradient at that point is defined according to the original diffusion curve model as follows:

[0139]

[0140] in and This represents the interpolated color value at this sampling point from the left and right color sources generated in step three. This definition reflects the instantaneous transition of color along the normal direction of the curve and serves as the basis for the Poisson-type diffusion equation. The right-hand term drives the diffusion of color in the image domain. The projection of the curve onto the pixel grid is labeled as... The pixels are considered as Dirichlet boundaries; and The region corresponds to the pixel to be solved within it. The gradient fields of the RGB three channels can also be constructed and averaged separately for each channel, so that the gradient driving terms of the final input curve rendering network have a unified form.

[0141] Finally, the gradient field Boolean mask Combined with RGB boundary colors 3D input tensor This serves as the input to the curve rendering network. The process remains globally differentiable, allowing control points, segment count, and color to all obtain inverse gradient signals from the final reconstruction error, thus enabling direct optimization of the curve geometry.

[0142] Step 5: Based on the input data constructed in Step 4, build a Fourier Neural Operator (FNO) curve rendering network. Figure 6 A neural approximation is used to solve the Poisson equation driven by the diffusion curve, generating a high-fidelity reconstructed image. The image is defined in a two-dimensional pixel domain. The position of any pixel is indicated by the symbol. It indicates that its color field is marked as (Corresponding to RGB components). Color gradient field determined based on diffusion curve. and fixed boundary color The rendering of the diffusion curve satisfies the following static Poisson equation:

[0143]

[0144]

[0145] in For the Laplace operator, The divergence of the color gradient field is defined by the set of pixels whose mask is set to 1, as given in step four.

[0146] To address this problem on a discrete pixel grid, the continuous field is discretized into a grid of size . The regular grid (in this embodiment, Construct the input tensor:

[0147]

[0148] in For fixed boundary mask, Border color The components on the three RGB channels.

[0149] In its implementation, the FNO curve rendering network first processes the input tensor... By concatenating normalized grid coordinates and applying a 1×1 convolutional channel blending layer with a linear mapping, the channel dimension is increased to a fixed width. To form the initial characteristic field:

[0150]

[0151] Subsequently, the characteristic fields pass through sequentially Iterative updates are performed on the Fourier blocks of any layer. have:

[0152]

[0153] in It is a channel-wise nonlinear activation function (GELU is used in this invention). This is a local linear mapping achieved through 1×1 convolution. and These represent the two-dimensional fast Fourier transform and its inverse transform of the feature field in the spatial dimension, respectively, and their corresponding architectures. Figure 5 The FFT and IFFT modules in the text This embodiment defines a learnable complex weight tensor (frequency domain kernel) in the truncated frequency domain, only... The value is taken in the truncated low-frequency region, symbol " "" represents a linear combination in the frequency domain channel dimension. This update form can be viewed as an iterative combination of a global integral operator implemented in the frequency domain and a local linear operator in the spatial domain, thereby approximating the input in a data-driven manner. To solve Operators .

[0154] The final feature field is obtained after several layers of Fourier blocks. Channel dimensionality reduction is achieved through two layers of linear projection to obtain the predicted image:

[0155]

[0156] in This represents the parameters of the FNO network.

[0157] The rendering process is also fully differentiable, ensuring that the difference between the reconstructed image and the input image can be fed back to the curve control points and segment prediction network, thereby forming a closed-loop optimization path of image-curve-image during the training phase and realizing end-to-end neural diffusion curve vectorization learning.

[0158] Step six involves a two-stage training strategy to collaboratively optimize the curve extraction network and the diffusion curve rendering network based on Fourier neural operators. This ensures that the resulting model simultaneously satisfies geometric accuracy, segment structure rationality, and diffusion physical consistency. In the training process, the geometric prediction network and the curve rendering network are first pre-trained separately to stabilize the initialized parameters. Then, end-to-end joint optimization is performed on this basis to allow the diffusion reconstruction error to drive the geometric prediction network in reverse.

[0159] The training datasets used by both networks are automatically constructed by a real diffusion curve generation system. Specifically, firstly, under a given image size constraint (in this embodiment...), According to a preset parameter range, multiple non-intersecting Bézier multi-segment curves are randomly sampled, including the number of segments (1-3 segments) per curve, the number of control points (4 control points per segment), and the number of curves (per frame). Figures 2-5 (Strips). Subsequently, based on the geometric control point parameters obtained from the sampling, a high-precision diffusion equation numerical solver was used to generate the corresponding ground truth color diffusion images, and the control point sequence, segment number label, and visibility label of each curve were recorded simultaneously, ultimately resulting in a synthetic dataset of 50,000 images.

[0160] The first stage is the pre-training stage. The geometric prediction network is trained on real diffusion curve data using the supervision signal. To enhance stability, Hungarian matching is used to establish a one-to-one correspondence between the predicted curve and the real curve in each input sample. For any real curve index... With arbitrary prediction curve index The matching cost is defined as:

[0161]

[0162] in For the first The existence prospect probability of the prediction curve is set. , . and They represent the first The prediction curve and the first The set of discrete sampling points of the true curve. The Chamfer distance is bidirectional and its expression is:

[0163]

[0164] After obtaining the cost matrix, the optimal match is obtained by solving an optimization problem that minimizes the cost:

[0165]

[0166] in, Indicates all lengths of The set of permutations and combinations, for each true curve index , This represents the index of the predicted curve that matches it. The optimal permutation is obtained through... Establish a set of matching pairs The loss function after matching can then be supervised element by element.

[0167] After obtaining the matching set, a supervision objective is constructed for the curve extraction network, consisting of existence loss, segment loss, coordinate loss, curve shape loss, and G1 continuity loss. The total loss is defined as:

[0168]

[0169] Among them, settings , , , , .

[0170] Specifically, the existence loss is expressed as a binary classification cross-entropy form:

[0171]

[0172] in, This represents the total number of queries. Indicates the first A query in real labels The corresponding predicted probabilities and class weights satisfy the following conditions: , The weights are for the background class. The coordinate loss uses the L1 norm form.

[0173]

[0174] in and These are predicted and actual multi-segment Bézier control point sequences, respectively. In the implementation, different weights are applied to endpoint and intermediate control points to strengthen the constraint on endpoint positions. The curve shape loss uses Chamfer distance based on multi-segment Bézier sampling.

[0175]

[0176] in and To utilize the cubic Bessel formula in step two The point set is obtained by segmenting and sampling multiple control points.

[0177] For multi-segment curves, an additional segment prediction loss is introduced, which takes the form of multi-class cross-entropy:

[0178]

[0179] in To predict the actual number of segments The probability of this. Meanwhile, to ensure the G1 continuity of multiple Bézier curves at the junctions between segments, a consistency loss based on the tangent direction is introduced:

[0180]

[0181] in The total number of connection points between segments in all matching pairs. and The first The first prediction curve The tangent direction vectors of the preceding and following segments at the connection points between segments.

[0182] Through the above loss combination, the geometric prediction network can learn multi-level constraints such as curve existence, segment selection, control point location, overall shape, and geometric continuity between segments during the pre-training stage.

[0183] The optimization process of the curve extraction network uses the AdamW optimizer to update the parameters, with the initial learning rate set to... It also incorporates a cosine annealing strategy to gradually decay the learning rate to the minimum during training. To improve convergence stability, a batch size of 16 was used, with the training and validation sets divided in an 8:2 ratio. Parallel loading was enabled during data loading to enhance training efficiency. During training, gradient clipping was applied to the parameters (typically with a maximum norm of 1.0) to maintain optimization stability. Model performance was evaluated on the validation set after each epoch, and the optimal weights were saved. The pre-training process lasted 20 epochs. After the curve extraction network had stably converged on the existence, segment number, and control point regression tasks, the end-to-end stage began, introducing reconstruction loss to further optimize the consistency between curve geometry and diffusion rendering solutions.

[0184] In the same phase, the curve rendering network is independently pre-trained based on real color gradient field data to learn the static solution capability of the diffusion equation, and obtain an operator approximation with convergent parameters. For input Generated Predicted Image The rendering network minimizes the following composite loss:

[0185]

[0186] in, This provides a high-precision numerical solution corresponding to the same diffusion curve parameters. This represents the sum of squares over all pixels and color channels. This is a discrete Laplacian approximation using finite difference. Specifically, the weighting coefficients for each loss term are set as follows: , , The first term in the formula guarantees overall pixel-level reconstruction accuracy, while the second term explicitly constrains the network output to satisfy the Dirichlet boundary conditions at fixed boundaries. The third point encourages predicted solutions to be spatially smooth and conform to the physical properties of diffusion processes. Subsequently, during the end-to-end joint training phase, all parameters of the network are frozen, serving only as a subset of the input. Differentiable operators are used to map curve geometric control points and color control points to the reconstructed image, so that the reconstruction error inverse constraint geometric prediction network generates diffusion curve parameters that better fit the solution of the diffusion equation.

[0187] The optimization process for the curve rendering network also uses the AdamW optimizer to update all network parameters, with the initial learning rate set to... It also incorporates a cosine annealing strategy to gradually decay the learning rate to the minimum during training. This ensures the overall smoothness and convergence stability of the optimization process. The batch size is set to 8, and the weight decay coefficient is... The gradient is clipped to a maximum norm of 1.0 before each parameter update. The training set contains 8000 samples, and the validation set contains 1000 samples. All samples are generated from the real diffusion curve parameters through high-precision PDE numerical calculation to produce corresponding diffusion images, and their color gradient fields, boundary masks, and boundary colors are constructed accordingly.

[0188] The second stage is the end-to-end reconstruction training stage. In this stage, the geometric prediction network and the curve rendering network are cascaded together to form a closed-loop structure of "image-curve-image". It no longer relies on the real diffusion curve annotation, nor does it impose geometric supervision constraints on the model such as existence, number of segments, coordinates, shape, or continuity. Instead, it constructs the end-to-end optimization objective entirely based on the pixel-level consistency between the reconstructed image and the input image. At this time, the curve rendering network remains frozen and only acts as a differentiable operator, mapping the gradient field and boundary color conditions generated by the predicted curve to the reconstructed image.

[0189] In this stage, the input image is first fed into a geometric prediction network to obtain predicted multi-segment diffusion curves. The geometric control points, number of segments, and existence of the curves are automatically learned and generated by the network. Subsequently, according to the process of steps three and four, the predicted curves are transformed into corresponding gradient fields. Fixed boundary mask and boundary color tensor The image is then fed as input into the curve rendering network, which has already been pre-trained in the first stage. The predicted reconstructed image is obtained through frequency domain convolution and spatial blending mechanisms. This stage aims to drive joint adaptive correction between the geometric and rendering networks through image-level errors. Therefore, the end-to-end objective function consists of only one pixel-level L1 loss, in the form of:

[0190]

[0191] in The reconstructed image obtained by FNO rendering. For the input image, This represents the L1 norm for each pixel.

[0192] Considering that the reconstruction loss has a significant impact on the network's gradient, in order to improve training stability, this stage is activated only after the geometric prediction network in the first stage has converged. This allows the network to further refine the control point positions and inter-segment relationships based on a stable geometric representation, thereby achieving synergistic optimization between the diffusion curve geometry and the diffusion rendering solution. Ultimately, this results in a more accurate vector representation of the target diffusion curve in terms of geometric consistency, color transition, and overall visual reconstruction.

[0193] Step 7: Based on the end-to-end optimized neural network model from Step 6, and driven by the input image, the final output is a high-quality vector image composed of several optimized diffusion curves. Figure 7 ).

[0194] Corresponding to the aforementioned embodiment of an image vectorization method based on neural diffusion curves, the present invention also provides an embodiment of an image vectorization device based on neural diffusion curves.

[0195] See Figure 8 The image vectorization device based on neural diffusion curves provided in this embodiment of the invention includes a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an image vectorization method based on neural diffusion curves in the above embodiment.

[0196] An embodiment of the image vectorization device based on neural diffusion curves provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 8 The diagram shown is a hardware structure diagram of any data processing device, including the image vectorization device based on neural diffusion curves provided by this invention. (Except for...) Figure 8In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0197] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an image vectorization method based on neural diffusion curves as described in the above embodiments.

[0198] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0199] It should be noted that in the claims and specification of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. While the invention has been illustrated and described with reference to certain preferred embodiments, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims

1. An image vectorization method based on neural diffusion curves, characterized in that, The method includes the following steps: Step 1: Acquire the input raster image; Step 2: Construct a curve extraction network based on an encoder-decoder architecture to predict several feature curves in the raster image and represent them in piecewise cubic Bézier form; Step 3: Based on the predicted curve, perform differentiable color sampling on both sides of the original image to construct a diffusion curve containing geometric control points and color control points; Step 4: Calculate the color gradient field based on the diffusion curve, and use the Boolean mask that identifies the fixed boundary points and the color components of the boundary points as input to the curve rendering network. Step 5: Construct a curve rendering network based on Fourier neural operators, perform neural approximation solution on the Poisson equation driven by the diffusion curve, and generate the reconstructed image. Step six, adopt a two-stage training strategy: the first stage supervises the training of the curve extraction network and the curve rendering network; the second stage performs end-to-end joint optimization of the curve extraction network based on the reconstruction loss. Step 7: Obtain the final vector image based on the neural diffusion curve using the trained curve extraction network and curve rendering network.

2. The image vectorization method based on neural diffusion curves according to claim 1, characterized in that, In step two, the curve extraction neural network includes a convolutional encoder and a multi-layer attention-based decoding structure; wherein, the convolutional encoder is used to process the input image. Generate multi-scale feature maps and fuse them to form sequence features. The decoding structure uses learnable query vectors as an implicit representation of the curve: ； in, For the initial query vector set, For the first Query vectors, with dimensions of . , The number of query vectors; the query vectors are updated at each layer through self-attention and cross-attention mechanisms, satisfying: ； in Indicates the first The layer decoder's update operator, based on the updated query vector, outputs the existence probability of each curve in the curve prediction head. Number of segments and control point coordinates ,in Indicates the first The first curve Coordinates of control points , This represents the maximum number of segments in the curve.

3. The image vectorization method based on neural diffusion curves according to claim 2, characterized in that, In step two, each predicted curve is represented by multiple cubic Bézier curves, where each curve contains the number of segments predicted by the network. And splice the segments inside the curve; each segment of the curve consists of four control points. The determined geometry satisfies the equation of a cubic Bézier curve: ； in, For curve parameters; for two adjacent segments within the same curve and The connection points satisfy the G1 continuity constraint, that is: ； ； The geometric structure of a single complete diffusion curve is formed by continuously splicing together multiple Bezier segments.

4. The image vectorization method based on neural diffusion curves according to claim 1, characterized in that, In step three, the set of sampling points is obtained by parametrically sampling the piecewise Bézier curves obtained in step two. and its corresponding curve parameters Where N is the number of sampling points; simultaneously, a tangent vector is calculated for each sampling point: ； And denoted at discrete sampling points ; Rotate the tangent vector counterclockwise around the origin. Obtain the normal vector: ； Using the normal vector to offset left and right distances within the image domain Obtain the left and right sampling points, where Indicates the fuzzy radius of the curve. Indicates the preset scaling factor: ； The left and right colors are obtained at the above positions using differentiable bilinear interpolation: ； and color sequence and Differentiable aggregation yields a fixed-dimensional sequence of left and right color control points. and and the curve control points predicted in step two. The parameter structure that together constitutes the diffusion curve .

5. The image vectorization method based on neural diffusion curves according to claim 4, characterized in that, In step four, the multiple diffusion curves obtained in step three are resampled to obtain a series of curve sampling points. Simultaneously, based on the left and right color control point sequences obtained in step three, in each resampling parameter The left and right color values ​​can be obtained by differentiable interpolation. and The corresponding unit normal vector is calculated. Construct a color gradient field at the curve position: ； And zeros are applied to pixels outside the curve, so that the color gradient only jumps near the curve; The projection of the curve onto the discrete pixel grid is marked as a Dirichlet boundary, forming a Boolean mask: ； in This represents a pixel position in the discrete pixel domain, where its coordinates are pairs of two-dimensional integers. ,when At that time, the pixel is located on the projected boundary of the diffusion curve; the boundary color component is assigned at the position satisfying the Dirichlet boundary. ; Finally, the gradient field Boolean mask and boundary color Composition forms a tensor: ； With tensor As input to the curve rendering network.

6. The image vectorization method based on neural diffusion curves according to claim 5, characterized in that, In step five, the curve rendering network first processes the input tensor... Concatenated with normalized grid coordinates, and mapped to an initial feature field of fixed width via a channel blending layer. Subsequently, the initial feature field passes through multiple layers of Fourier operator modules, and in each layer, it is updated by combining local spatial domain mapping with global frequency domain operations. The update form is expressed as: ； in It is a channel-wise nonlinear activation function. It is a local linear mapping. and These are the two-dimensional Fast Fourier Transform and its inverse transform, respectively. A learnable frequency domain kernel defined on the truncated frequency domain; This represents the feature field after the (t+1)th update; pass Layer-by-layer Fourier block iterations, using a rendering network based on the Fourier Neural Operator (FNO) to approximate the input tensor in a data-driven manner. To the solution of the diffusion equation The mapping operator in the final feature field The predicted image is generated by linear projection. .

7. The image vectorization method based on neural diffusion curves according to claim 1, characterized in that, In step six, the supervised training of the curve extraction network and the curve rendering network in the first stage of the two-stage training is based on the matching relationship between the predicted curve and the real curve. Hungarian matching is used to establish a one-to-one correspondence between the predicted curve and the real curve in each input sample. For any real curve index... With arbitrary prediction curve index The matching cost is defined as: ； in For the first The existence probability of the prediction curve and For balance coefficient, This is the bidirectional Chamfer distance. and They represent the first The prediction curve and the first The set of discrete sampling points for the true curves; after obtaining the cost matrix, the optimal match is obtained by solving an optimization problem that minimizes the cost: ； in, Indicates all lengths of The set of permutations and combinations, for each true curve index , Indicates the index of the predicted curve that matches it; through the obtained optimal permutation Establish a set of matching pairs The loss function after matching can then be supervised element by element.

8. The image vectorization method based on neural diffusion curves according to claim 7, characterized in that, In step six, the curve extraction network in the first stage of the two-stage training undergoes supervised training based on the annotation of the real diffusion curve, and its loss function includes the following five terms: ； Among them, the items preceding each loss item Indicates the weight of the corresponding item; Existential loss Using the binary classification cross-entropy form: ； in This represents the total number of queries. Indicates the first A query in real tags The corresponding predicted probabilities and class weights satisfy the following conditions: , Weights for background classes; coordinate loss Using L1 norm form: ； in and These are the predicted and actual multi-segment Bezier control point sequences, respectively. In the implementation, different weights are applied to the endpoint control points and intermediate control points to strengthen the constraint on the endpoint positions. Curve shape loss Chamfer distance based on multi-segment Bezier sampling is used: ； in and To utilize the cubic Bessel formula in step two The point set obtained by segmenting and sampling multiple control points; For multi-segment curves, an additional segment prediction loss is introduced. It adopts a multi-class cross-entropy form: ； in To predict the actual number of segments The probability of this; simultaneously, to ensure the G1 continuity of multiple Bézier curves at the junctions between segments, a consistency loss based on the tangent direction is introduced. : ； in The total number of connection points between segments in all matching pairs. and The first The first prediction curve The tangent direction vectors of the preceding and following segments at the connection points between segments.

9. The image vectorization method based on neural diffusion curves according to claim 7, characterized in that, In step six, the curve rendering network in the first stage of the two-stage training is independently pre-trained based on real color gradient field data to minimize the following composite loss: ； in, This represents the weighting coefficient for each loss term; This provides a high-precision numerical solution corresponding to the same diffusion curve parameters. This represents the sum of squares over all pixels and color channels. This is a discrete Laplacian approximation using finite difference; the first term guarantees overall pixel-level reconstruction accuracy, and the second term explicitly constrains the network output to satisfy Dirichlet boundary conditions at fixed boundaries. The third point encourages predicted solutions to be spatially smooth and conform to the physical characteristics of diffusion processes; In the second stage of the two-stage training strategy, the curve prediction network and the curve rendering network are connected end to end to form a closed-loop differentiable structure of "image-curve-image" from the input image to the predicted curve and then to the reconstructed image. During this stage, the curve rendering network remains frozen and is used only as a differentiable operator. The end-to-end optimization objective is constructed based on the pixel-level consistency between the reconstructed image and the input image, and the optimization loss is in the form of: ； in For the input image, This is the reconstruction result obtained by freezing the rendering network.

10. An image vectorization device based on neural diffusion curves, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that... When the processor executes the executable code, it implements an image vectorization method based on neural diffusion curves as described in any one of claims 1-9.

Images