A diffusion model-based text repair method and device based on double-condition guidance

Abstract

The application discloses a diffusion model character repairing method and device based on double-condition guidance, and belongs to the field of artificial intelligence in the character repairing technology, and the method comprises the steps of obtaining a first feature matrix and a second feature matrix; taking the first feature matrix as a query vector, and performing cross-attention fusion on the key vector and the value vector of the second feature matrix to generate a target condition vector; injecting the target condition vector into a noise preset network of a diffusion model with an initial input of a character image to be repaired, dynamically fusing the local features of the skip connection with the target condition vector through a cross-attention block to generate enhanced features; and then combining the weight scheduling of the skeleton prior and the stroke prior to generate a repaired character image. The structure and stroke features are obtained through two feature matrices respectively, the character constraint condition is dynamically introduced in the reverse denoising process of the diffusion model, so that the model can generate a high-quality repairing result while retaining the details of the strokes and the character style features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence of text restoration technology, and more particularly, to a diffusion model text restoration method and device based on double condition guidance. BACKGROUND

[0002] As an important carrier of ancient Chu culture, Chu-style characters contain rich historical, cultural and academic value. However, due to the fact that they have been passed down for thousands of years, these characters have been damaged to varying degrees by natural factors such as water and wind, resulting in missing strokes and blurred characters. In this case, restoration usually involves a highly complex, time-consuming and professional workflow.

[0003] The existing technology in the field of text restoration mainly includes traditional text restoration methods, image generation methods based on deep learning, and diffusion models.

[0004] Traditional text restoration methods achieve restoration through stroke feature extraction, edge detection, and image segmentation techniques. However, they have limited effectiveness in handling complex backgrounds, severely degraded images, and missing detail strokes, and are prone to structural distortion (such as stroke breakage and adhesion) or semantic errors (such as incorrect combination of Chinese character radicals). Moreover, the restoration process lacks flexibility. Existing condition control methods based on text prompts are difficult to accurately constrain the structural details of the text and lack robustness for complex characters or low-quality inputs.

[0005] In image generation methods based on deep learning, VAE-based restoration methods can lack detail control. GAN-based restoration methods are susceptible to instability in adversarial training, rely on a large amount of labeled data, and are prone to blurry or artifact restoration strokes, making it difficult to constrain the global structure. CNN-based restoration methods rely on local textures and lack explicit modeling of the global skeleton and stroke order of the text.

[0006] Diffusion models have shown outstanding performance in image generation, but the prior mask resolution in existing technology limits the fine restoration capability, the restoration process lacks dynamic adjustment, and it is difficult to accurately control the stroke direction and skeleton topology, leading to structural distortion of the generated text. At the same time, there is a certain dependence on training data. SUMMARY

[0007] The embodiments of the present application provide a diffusion model text restoration method and device based on double condition guidance to solve the problem of insufficient restoration accuracy and adaptability to complex damage scenarios in the text restoration process of existing technology.

[0008] A diffusion model text restoration method based on double condition guidance, comprising:

[0009] obtaining a first feature matrix and a second feature matrix; the first feature matrix is a global skeleton feature matrix obtained by modeling a preset complete character image based on a text skeleton topology; and the second feature matrix is a stroke timing feature matrix obtained by modeling a preset complete character image based on timing features and morphological details;

[0010] taking the first feature matrix as a query vector, performing cross-attention fusion on a key vector and a value vector of the second feature matrix, and generating a target condition vector;

[0011] injecting the target condition vector into a noise preset network of a diffusion model with an initial input being a to-be-repaired character image, dynamically fusing local features connected by a skip connection with the target condition vector through a cross-attention block, and generating an enhanced feature;

[0012] linearly combining the enhanced feature with the first feature matrix and the second feature matrix, and combining weight scheduling of skeleton prior and stroke prior to generate a repaired character image.

[0013] Further, the taking the first feature matrix as a query vector, performing cross-attention fusion on a key vector and a value vector of the second feature matrix, and generating a target condition vector comprises:

[0014] projecting the first feature matrix to a query space, and projecting the second feature matrix to a key space and a value space respectively;

[0015] calculating a similarity matrix of the query space and the key space, and normalizing by using a Softmax function to obtain an attention weight;

[0016] weighting and summing the value vector based on the attention weight to obtain a target condition vector.

[0017] Further, the injecting the target condition vector into a noise preset network of a diffusion model with an initial input being a to-be-repaired character image, dynamically fusing local features connected by a skip connection with the target condition vector through a cross-attention block, and generating an enhanced feature comprises:

[0018] at a skip connection of the noise preset network of the diffusion model, taking the first feature matrix as a query, and matching the key vector and the value vector of the second feature matrix;

[0019] calculating a context vector by scaling dot product attention, and generating an enhanced feature by convolution fusion after splicing the preset complete character image.

[0020] Further, before the combining weight scheduling of skeleton prior and stroke prior to generate a repaired character image, further comprising:

[0021] In the reverse denoising process of the diffusion model, a dynamic stroke generation mechanism is introduced, and the stroke area to be repaired is activated in stages through a noise mask;

[0022] The constraint formula of the noise mask is:

[0023] The constraint formula of the noise mask is:

[0024] (11);

[0025] Wherein, is the stroke area mask to be repaired in the current stage; through Hadamard product, the denoising is strengthened in the mask area, and the original prediction is kept in the non-mask area;

[0026] That is , represents the new noise prediction, which is the updated result obtained after the network repairs the stroke area more strongly in the current step.

[0027] , represents the original noise prediction, that is, the noise estimate output by the model before the special processing of the stroke area.

[0028] , represents the final output noise prediction, which uses the new prediction value in the stroke area and keeps the original prediction value in the non-stroke area.

[0029] When , it represents the stroke part that needs to be repaired, and the model is forced to apply the new predicted noise Denoising is performed on the specified stroke area;

[0030] When , the noise state of the last time step is reserved for the background part other than the text, avoiding interference with the repaired or unrepaired areas;

[0031] The stage-by-stage activation is a stage-by-stage conditional weight scheduling through a dynamic weight formula;

[0032] The dynamic weight formula is specifically:

[0033] ;

[0034] In the early stage, t→T, high noise stage,

[0035] ; skeleton feature dominant, the model preferentially restores the global topological structure of Chinese characters, ensuring the standardization of the character shape.

[0036] In the middle stage, t=T / 2, transition stage,

[0037] ; The skeleton feature weight gradually decreases, and the stroke feature starts to participate in the local shape optimization.

[0038] In the late stage, t→0, low noise stage,

[0039] ; Stroke feature dominates, model focuses on stroke sharpening, ink density, etc. Detail reconstruction, improve visual authenticity.

[0040] Further, the first feature matrix acquisition step includes:

[0041] Input the preset complete character image into the skeleton graph encoder, output the node feature matrix representing the global skeleton topology, and obtain the first feature matrix; The skeleton graph encoder is constructed based on graph attention network.

[0042] Further, the implementation of the skeleton graph encoder includes:

[0043] Extract the binary skeleton image based on the preset complete character image;

[0044] Model the binary skeleton image as a graph structure G=(V,E);

[0045] Wherein, the node set V represents the intersection of strokes, and the edge set E represents the connection relationship of strokes;

[0046] Update the node feature to the global skeleton feature matrix by aggregating the node neighborhood features through the multi-layer graph attention network.

[0047] Further, the second feature matrix acquisition step includes:

[0048] Input the preset complete character image into the stroke set encoder, output the feature matrix encoding the local stroke shape and timing dependence, and obtain the second feature matrix; The stroke set encoder is constructed by the Transformer encoder.

[0049] Further, the implementation of the stroke set encoder includes:

[0050] Based on the preset complete character image, extract the stroke direction encoding sequence arranged in writing order to obtain the normalized stroke direction vector sequence;

[0051] Input the normalized stroke direction vector sequence, and introduce the sine position coding to reserve the writing timing information;

[0052] Calculate the self-attention weight through the multi-layer Transformer encoder, and output the stroke timing feature matrix.

[0053] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory having stored therein computer program instructions which, when executed by the processor, cause the processor to perform the diffusion model character repairing method based on double-condition guidance as described above.

[0054] According to still another aspect of the present application, a computer readable storage medium is provided, having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the diffusion model character repairing method based on double-condition guidance as described above.

[0055] The embodiment of the present application provides a diffusion model character repairing method and device based on double-condition guidance, and the method comprises the following steps: acquiring a first feature matrix and a second feature matrix; wherein the first feature matrix is a global skeleton feature matrix obtained by modeling a character skeleton topology based on a preset complete character image; and the second feature matrix is a stroke timing feature matrix obtained by modeling timing features and morphological details based on the preset complete character image. The first feature matrix is taken as a query vector, cross-attention fusion is performed on the query vector, a key vector and a value vector of the second feature matrix, and a target condition vector is generated; the target condition vector is injected into a noise preset network of a diffusion model with an initial input being a character image to be repaired, local features connected by a skip connection are dynamically fused with the target condition vector through a cross-attention block, and enhanced features are generated; and the enhanced features are linearly combined with the first feature matrix and the second feature matrix, and a repaired character image is generated by combining weight scheduling of skeleton priori and stroke priori.

[0056] The present application obtains structure and stroke features through a global skeleton feature matrix and a stroke timing feature matrix respectively, adopts cross-attention fusion of the two kinds of priori at the skip connection, obtains a target condition vector, replaces original features of the network with the target condition vector, then performs character constraint condition injection, and further restores missing strokes through a step-by-step iteration process, so that the clear and complete character image is generated while the image denoising is completed, so that the model can generate high-quality repair results while retaining stroke details and character style features. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0058] Figure 1 The flowchart of the diffusion model character repairing method based on double-condition guidance provided by the embodiment of the present application is shown in the figure.

[0059] Figure 2 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0061] The terms "first", "second", "third", etc. in the specification and claims of the present application and the above-described drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units that are not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0062] Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.

[0063] SUMMARY

[0064] The prior art in the field of text restoration mainly includes traditional text restoration methods, image generation methods based on deep learning, and diffusion models.

[0065] Traditional text restoration methods achieve restoration through techniques such as stroke feature extraction, edge detection, and image segmentation, and perform well in slightly damaged images, but have limited effect in handling complex backgrounds, severely degraded images, and missing detail strokes, and are prone to structural distortion (such as stroke breakage, sticking) or semantic errors (such as incorrect combination of Chinese character components), and lack flexibility in the restoration process. Existing conditional control methods based on text prompts are difficult to accurately constrain the structural details of the text, and lack robustness for complex characters or low-quality inputs.

[0066] In the image generation method based on deep learning, the repair method based on VAE can generate natural repair samples, but lacks detail control. The repair method based on GAN can generate real repair images, but the generation result is easily affected by the instability of adversarial training, relies on a large amount of labeled data, and the repair strokes are prone to blur or artifacts, and it is difficult to constrain the global structure. The repair method based on CNN relies on local texture and lacks explicit modeling of the global skeleton and stroke sequence of the text.

[0067] The diffusion model performs outstandingly in the field of image generation, but the prior mask resolution in the prior art limits the fine repair capability, the repair process lacks dynamic adjustment, and the stroke direction and skeleton topology cannot be accurately controlled, resulting in distortion of the generated text structure, and there is a certain dependence on training data.

[0068] To solve the above technical problems, the present application proposes a diffusion model text repair method and device based on double condition guidance, aiming to solve the problems of insufficient detail restoration, poor semantic consistency and structural distortion and stroke detail loss in degraded text repair of traditional text repair technology in complex scenes, and improve the structural rationality and visual coherence of the generated result. The core of the present application is to constrain the reverse denoising process of the diffusion model through the double condition guidance mechanism (skeleton topology and stroke timing), so as to balance the global structural rationality and local stroke detail accuracy when repairing the text image.

[0069] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be specifically introduced with reference to the accompanying drawings.

[0070] Exemplary method

[0071] Figure 1 Fig. 1 illustrates a flowchart of a diffusion model text repair method based on double condition guidance according to an embodiment of the present application.

[0072] As Figure 1 shown, the diffusion model text repair method based on double condition guidance according to an embodiment of the present application comprises:

[0073] S110, obtaining a first feature matrix and a second feature matrix;

[0074] The first feature matrix is a global skeleton feature matrix obtained by modeling the text skeleton topology based on a preset complete character image; the second feature matrix is a stroke timing feature matrix obtained by modeling the timing feature and morphological detail based on the preset complete character image;

[0075] S120, taking the first feature matrix as a query vector, and performing cross-attention fusion with the key vector and the value vector of the second feature matrix to generate a target condition vector;

[0076] S130, inject the target condition vector into a noise preset network of a diffusion model with an initial input of a character image to be repaired, dynamically fuse the local features of the skip connection with the target condition vector through a cross attention block, and generate enhanced features;

[0077] S140, linearly combine the enhanced features with the first feature matrix and the second feature matrix, and combine the weight scheduling of the skeleton prior and the stroke prior to generate a repaired character image.

[0078] Skeleton and stroke feature extraction can model the skeleton topology of the text through a skeleton graph encoder (SGE), and model the stroke timing feature through a stroke set encoder (SSE).

[0079] Cross-modal condition fusion can fuse the skeleton feature and the stroke feature into a condition vector through a cross-attention mechanism.

[0080] Condition injection and dynamic generation are to embed the condition vector into the U-Net architecture of the diffusion model, and combine dynamic weight scheduling to repair missing strokes in stages.

[0081] In the following, each step will be described in detail.

[0082] In step S110, the first feature matrix and the second feature matrix are obtained.

[0083] The first feature matrix is a global skeleton feature matrix obtained by modeling the skeleton topology of the text based on a preset complete character image; and the second feature matrix is a stroke timing feature matrix obtained by modeling the timing feature and the morphological details based on the preset complete character image.

[0084] For example, the preset complete character image is input into a skeleton graph encoder, and a node feature matrix representing the global skeleton topology is output, obtaining the first feature matrix; the skeleton graph encoder is constructed based on a graph attention network.

[0085] Further, the implementation of the skeleton graph encoder includes:

[0086] Extract a binary skeleton image based on a preset complete character image;

[0087] Model the binary skeleton image as a graph structure G=(V,E);

[0088] Wherein, the node set V represents the stroke intersection, and the edge set E represents the stroke connection relationship.

[0089] Update the node features to the global skeleton feature matrix by aggregating the node neighborhood features through a multi-layer graph attention network.

[0090] Specifically, the Canny edge detection algorithm can be used to extract the character contour to generate an initial binary image; a thinning algorithm (such as the Zhang-Suen algorithm) is applied to the binary image to obtain a single-pixel-width skeleton graph.

[0091] The skeleton graph is modeled as a graph structure G = (V, E), where the node set V = {v1, v2,..., vN} represents the stroke intersection, and the edge set E = {e1, e2,..., eN} represents the stroke connection relationship. The network uses a graph attention network to capture multi-scale spatial dependencies between nodes through multi-layer feature aggregation. The network uses a graph attention network to capture multi-scale spatial dependencies between nodes through multi-layer feature aggregation.

[0092] First, initialize the node features, and the initial feature of each node v is generated by linear projection from its spatial coordinates (x, y).

[0093] At the Lth layer, the neighborhood features are aggregated, and the feature of node vi is updated to , and the formula is as follows:

[0094] (1);

[0095] where N(v) is the neighbor node set, σ is the activation function, and W is the learnable parameter matrix. After passing through the L-layer graph attention network, the features of all nodes are updated, and the first feature matrix, i.e., the global skeleton feature matrix

[0096] is output, which represents the global skeleton topology for subsequent cross-modal fusion. (N represents the number of skeleton nodes), which represents the global skeleton topology for subsequent cross-modal fusion.

[0097] In an optional embodiment, the implementation of the skeleton graph encoder (SGE) includes network architecture and hierarchical division of labor.

[0098] In this embodiment, the skeleton graph encoder uses a four-layer graph attention network (GAT), and the number of neurons at each layer is {64, 64, 128, 128} in turn. The first 2 layers: focus on local feature aggregation, capture the spatial relationship of stroke intersections through neighborhood node feature interaction. The 3rd-4th layer: expand the range of action, capture the global topological relationship across nodes. For example, the 3rd layer maps 128-dimensional features to aggregate the dependency relationship between distant nodes, ensuring the global consistency of the skeleton topology.

[0099] After the global skeleton feature matrix is output, it can be mapped to a channel dimension (such as 512 dimensions) compatible with the diffusion model U-Net through a linear projection layer, as the query input for cross-attention fusion.

[0100] ​​​​​The interaction between skeleton features and diffusion models can be achieved by obtaining a binary skeleton graph, performing graph structure modeling, initializing node features, and then performing GAT layer feature aggregation to obtain a global skeleton feature matrix (e.g., N×128), which is then used as the query vector for cross-attention fusion.

[0101] Optionally, the steps for obtaining the second feature matrix include:

[0102] The pre-set complete character image is input into the stroke set encoder, and the output feature matrix that encodes the local stroke shape and temporal dependency is obtained to obtain the second feature matrix; the stroke set encoder is constructed by the Transformer encoder.

[0103] Furthermore, the implementation of the stroke set encoder includes:

[0104] Based on a pre-defined complete character image, a stroke direction encoding sequence arranged in the writing order is extracted to obtain a normalized stroke direction vector sequence.

[0105] Input a normalized sequence of stroke direction vectors and introduce sinusoidal position coding to preserve writing timing information;

[0106] The self-attention weights are calculated by multi-layer Transformer encoders, and the stroke temporal feature matrix is ​​output.

[0107] Specifically, firstly, the characters are decomposed into strokes based on the skeleton diagram and arranged in the order of writing. S ={ s 1, s 2,..., sM},in M Number of strokes;

[0108] Secondly, for each stroke sk Calculate its main direction (such as horizontal, vertical, left-falling, and right-falling strokes) and normalize it into a direction vector dk∈R4d k ∈R4, for example:

[0109] Horizontal stroke: d k =[1,0,0,0];

[0110] Vertical stroke: d k =[0,1,0,0];

[0111] The same logic applies to the other directions.

[0112] Then, the network is built based on a six-layer Transformer encoder, and sinusoidal position coding is introduced to preserve the timing information of stroke writing. The stroke features are concatenated and then input into the encoder. The attention weights of the k-th layer are calculated as follows:

[0113] (2)

[0114] where, are generated by linear projection of stroke features, and the attention dimension is .

[0115] After attention computation and fully connected layer, the final output is the second feature matrix, i.e., the stroke sequential feature matrix (M represents the number of strokes), which encodes the local stroke shape and sequential dependency.

[0116] In an optional embodiment, the stroke set encoder aims to convert the stroke sequence arranged in writing order into a high-dimensional feature representation by modeling the sequential features and shape details of Chinese character strokes. The input is the stroke direction encoding sequence S= where each stroke represents the normalized direction vector of horizontal, left, vertical, and dot, and the initial stroke feature matrix with dimension 64 is generated by linear projection .

[0117] In this embodiment, the stroke set encoder is constructed based on a six-layer Transformer encoder. The query (Q) of each stroke is calculated with the key (K) of all strokes to generate attention weights by Softmax.

[0118] Among them, the six-layer transformer is used to capture the stroke series dependency and avoid overfitting. It includes 1-2 shallow layers: learning the connection direction of adjacent strokes; 3-4 transition layers: cross-stroke temporal dependency; 5-6 deep layers: learning global sequence (writing rules of the entire Chinese character).

[0119] The six-layer Transformer encoder also contains four self-attention heads, each with the following functions:

[0120] Head 1 (direction similarity): calculate the cosine similarity of stroke direction vectors, for example, the attention weight of horizontal stroke is higher between strokes with similar directions; Head 2 (temporal proximity): capture the temporal dependency of adjacent strokes, such as the writing order of “horizontal” followed by “vertical”; Head 3 (shape complementarity): analyze the shape complementarity features of symmetric structures (such as the left and right strokes of the “mouth” character); Head 4 (cross-stroke structure constraint): model the spatial relationship of surrounding and semi-surrounding structures, such as the outer frame and internal strokes of the “country” character.

[0121] It is worth noting that the attention dimension of each head is , which gradually enhances the context representation of each stroke, making it contain both its own shape information and global sequence features. The self-attention mechanism cannot capture the positional relationship between strokes, so we introduce the sine position encoding , mainly responsible for stroke positioning. The periodicity of the sine function can capture the timing relationship of near and far distances at the same time to preserve the stroke writing timing information. The stroke feature is input into the encoder after splicing.

[0122]

[0123] wherein, are respectively generated by linear projection of stroke features.

[0124] Each layer of self-attention is followed by two fully connected networks with dimensions of 256 and 128, respectively. The first layer: latitude 64-256, expansion, learning more complex structure of the text; the second layer: 256-128, retaining key information and then facilitating the alignment of the final output features with The activation function is GELU, which enhances the non-linear expression ability.

[0125] After attention calculation and fully connected layer, the final output stroke timing feature matrix (M represents the number of strokes) encodes the local stroke shape and timing dependency.

[0126] After the output of the stroke timing feature matrix, the dimension can be adjusted through the fully connected layer (e.g. 128→512) as the key (K) and value (V) of the cross-attention module.

[0127] It is worth noting that, in order to eliminate the modal difference between the skeleton and the stroke, a spatial alignment strategy can be used in cross-attention fusion, such as mapping the skeleton node coordinates and stroke direction vectors to the same high-dimensional space. Or time alignment, through sine position encoding to preserve the stroke writing order, and dynamically match the static structure of the skeleton topology.

[0128] In step S120, the first feature matrix is taken as a query vector, and the key vector and value vector of the second feature matrix are cross-attention fused to generate a target condition vector.

[0129] In order to better capture the complementary features of skeleton and stroke in spatial topology and timing shape, and eliminate the modal difference between skeleton and stroke features, a cross-attention strategy can be used, which takes the first feature matrix and the second feature matrix as input features of different modalities, respectively projects them into query (Q), key (K) and value (V) space, takes the skeleton feature as the query, and matches the key and value of the stroke feature, thereby capturing the association between them:

[0130] (3)

[0131] wherein, A learnable projection matrix.

[0132] The correlation is calculated by dot product and normalized to obtain an attention weight, and the stroke value vector is weighted by the attention weight to generate a fused target condition vector.

[0133] Optionally, in the diffusion model-based character repairing method provided in the double-condition-guided diffusion model, the first feature matrix is taken as a query vector, cross-attention fusion is performed on the key vector and the value vector of the second feature matrix, and a target condition vector is generated.

[0134] The first feature matrix is projected into a query space, and the second feature matrix is projected into a key space and a value space, respectively.

[0135] A similarity matrix of the query space and the key space is calculated, and a Softmax function is used for normalization to obtain an attention weight.

[0136] The value vector is weighted and summed based on the attention weight to obtain a target condition vector.

[0137] Specifically, the skeleton feature matrix is taken as a query (Query), and the stroke feature matrix is taken as a key (Key) and a value (Value), and the two types of features are fused through a cross-attention mechanism. Wherein, the skeleton feature is projected linearly to generate a query vector Q , and the stroke feature is projected into a key vector K and a value vector V . After calculating the dot product similarity matrix of Q and K , the attention weight is generated through Softmax normalization. For example, if the number of skeleton nodes is 50 and the number of strokes is 8, the size of the attention weight matrix is 50x8, which represents the attention degree of each skeleton node to each stroke.

[0138] Subsequently, the value vector V is weighted and summed using the attention weight to generate a fused target condition vector. For example, a certain skeleton node may have a high weight (such as 0.8) for the key vector of the 3rd stroke, so the condition vector of the node will mainly fuse the morphological information of the 3rd stroke. This condition vector contains the topological constraints of the skeleton and the timing details of the stroke, and provides structured priori for the subsequent diffusion model.

[0139] In an optional embodiment, the cross-attention fusion of the first feature matrix as a query vector and the key vector and the value vector of the second feature matrix includes:

[0140] First, a linear projection layer is constructed to project the skeleton feature (Q) and the stroke feature (K / V) into a 512-dimensional space.

[0141] Secondly, the scaled dot-product attention calculation is performed, for example:

[0142] Attention = softmax )V

[0143] where dk=64 is the attention dimension, and the Softmax normalization generates a 50x8 weight matrix (skeleton node number x stroke number);

[0144] Then, the context vector generation is performed, and the value vector is weighted and summed to output the conditional vector .

[0145] In addition, the conditional vector can be fused with the original feature map F (64x64x256) at the third layer jump connection of the U-Net by, for example, feature replacement, and the conditional vector is fused into an enhanced feature map F_fused (64x64x512) through 1x1 convolution; and, for example, dynamic weight scheduling is adopted, for example, a cosine function is used to adjust the weight ratio of the skeleton and the stroke, and then the fused target conditional vector is obtained.

[0146] In step S130, the target conditional vector is injected into the noise preset network of the diffusion model with the initial input being the character image to be repaired, and the local features of the jump connection are dynamically fused with the target conditional vector through the cross attention block to generate enhanced features.

[0147] The image to be repaired is used as the initial input of the diffusion model, and gradually denoised in the reverse process, and the corresponding conditional vector is combined to generate a repair result.

[0148] In this embodiment, the jump connection of the noise preset network U-Net of the diffusion model can be directly transmitted from the feature map of a certain layer of the encoder (for example, a 4-layer down-sampling module, and the number of channels of each layer is {64, 128, 256, 512} in turn, and step 2 convolution and LeakyReLU activation are adopted) to the corresponding layer of the decoder (for example, a 4-layer up-sampling module, and the resolution is restored through bilinear interpolation and convolution, and the number of channels of each layer is symmetrical to the encoder), so as to retain multi-scale spatial information and retain high-resolution information. It should be noted that in this embodiment, the fusion condition C (external condition information) can provide high-level conditions about the overall structure and detail style of the target image.

[0149] Therefore, the cross attention block is introduced at the jump connection, C is taken as the key (Key) and the value (Value), and the local features F of the jump connection are taken as the query (Query), so as to realize dynamic fusion of information, so that the decoder can adaptively adjust the feature weight according to the prior information, and better guide the reconstruction process.

[0150] First, the features of the current skip connection are , the spatial resolution is , and the number of channels is D. At the same time, the external prior condition vector , the sequence length is N, and the feature dimension is d. Next, the encoder features and the condition vector are mapped to the query (Query), key (Key) and value (Value) spaces using three learnable linear mappings. The similarity between the encoder features (Query) and the condition vector (Key) is calculated using the scaled dot-product attention formula, and the attention weight A is generated by Softmax normalization:

[0151] (4)

[0152] where the weight represents the degree of dependence of the encoder feature at position (h, w) on the nth element of the condition vector.

[0153] The value (Value) of the condition vector is then weighted and summed using the attention weight to generate the context vector Context, which carries global prior information related to the original features. The context vector is fused with the original feature map, and then the original feature map and the reshaped context tensor are concatenated in the channel dimension to ensure that both parts of the information (local features and global priors) are retained. To automatically learn how to integrate the concatenated feature information, a 1*1 convolution layer can be used to fuse the concatenated result, and finally a new feature map is generated, as follows:

[0154] (5);

[0155] The generated not only contains the local detailed features of F, but also fully embeds the global context information obtained through the attention mechanism. The fused feature is passed to the corresponding layer of the decoder through the skip connection and concatenated or added with the decoder's upsampled features to drive detail restoration.

[0156] Optionally, in the diffusion model-based character repair method provided in the double-condition-guided diffusion model character repair method, the target condition vector is injected into the noise preset network of the diffusion model with the initial input being the character image to be repaired. The local features of the skip connection are dynamically fused with the target condition vector through a cross-attention block to generate enhanced features, including:

[0157] In the skip connection of the noise preset network of the diffusion model, the first feature matrix is taken as the query, and the key vector and the value vector of the second feature matrix are matched;

[0158] For example, the context vector is calculated by scaled dot-product attention, and the enhanced feature is generated by concatenating the context vector with the preset complete character image and then fusing them through convolution.

[0159] Specifically, in the noise prediction network of the diffusion model, the target condition vector is dynamically injected into the skip connection of the U-Net. Taking the third layer skip connection of the U-Net as an example, the local feature map output by the encoder has a size of 64x64x256, and the condition vector C is reshaped to the same spatial size and then fused with the local feature through a cross-attention block. Specifically, the local feature is taken as the query, and the condition vector C is taken as the key and value to calculate the scaled dot-product attention. For example, the query vector at a certain position (h, w) in the local feature is calculated with all the key vectors of the condition vector to generate an attention weight matrix, and then the value vectors are weighted and summed to obtain a context vector.

[0160] Finally, the context vector is concatenated with the original local feature, and the channel information is fused through a 1x1 convolution layer. For example, the original feature has 256 channels, and the context vector has 128 channels. After concatenation, the dimension is reduced to 256 channels through convolution to generate an enhanced feature map, and the enhanced feature is obtained to guide the detail reconstruction of the missing area.

[0161] In step S140, the enhanced feature is linearly combined with the first and second feature matrices, and the weight scheduling of the skeleton prior and stroke prior is combined to generate a repaired character image.

[0162] In this embodiment, the enhanced feature is introduced into the noise prediction network to introduce the condition information of the global skeleton feature matrix and the stroke timing feature matrix .

[0163] The improved Gaussian distribution can be:

[0164] (6);

[0165] Correspondingly, the mean formula can be changed to:

[0166] (7);

[0167] The variance remains unchanged:

[0168] (8);

[0169] Wherein the mean and variance of the conditional inverse distribution are determined by ; and the condition .

[0170] The sampling formula of each step of the inverse process can be:

[0171] (9);

[0172] By bringing in the mean expression, the :

[0173] (10);

[0174] Through the iterative process step by step, the missing strokes are recovered while the image denoising is completed, until the clear and complete repaired character image is generated.

[0175] Optionally, before the step of generating the repaired character image by combining the weight scheduling of the skeleton prior and the stroke prior, the method further includes:

[0176] In the reverse denoising process of the diffusion model, the noise mask mechanism is introduced, the noise prediction range is dynamically constrained, and the fine control of the stroke sequence of Chinese characters is enhanced.

[0177] The constraint formula of the noise mask is:

[0178] (11);

[0179] wherein, is the stroke area mask to be repaired in the current stage; by Hadamard product, the denoising is strengthened in the mask area, and the original prediction is kept in the non-mask area;

[0180] : that is , indicates the new noise prediction, which is the update result obtained after the network performs stronger repair on the stroke area in the current step.

[0181] , indicates the original noise prediction, that is, the noise estimate output by the model before the special processing of the stroke area.

[0182] , indicates the final output noise prediction, which uses the new prediction value in the stroke area and keeps the original prediction value in the non-stroke area.

[0183] When , indicates the stroke part that needs to be repaired, and the model is forced to apply the new predicted noise to denoise the specified stroke area;

[0184] When , the noise state of the last time step is reserved for the background part other than the character, so as to avoid interfering with the repaired or unrepaired areas.

[0185] The phased activation is a phased conditional weight scheduling through a dynamic weight formula.

[0186] The dynamic weight formula is specifically:

[0187] (12)

[0188] Early stage t→T, high noise stage,

[0189] The skeleton feature dominates, the model preferentially restores the global topological structure of Chinese characters, and ensures the standardization of the character shape.

[0190] Middle stage, t=T / 2, transition stage,

[0191] The skeleton feature weight gradually decreases, and the stroke feature starts to participate in the local shape optimization.

[0192] Late stage, t→0, low noise stage,

[0193] The stroke feature dominates, the model focuses on the reconstruction of details such as stroke sharpening and ink density, and improves the visual authenticity.

[0194] Optionally, a cosine scheduling function α(t) is used to control the weight of the skeleton feature, 1 α(t) is used to adjust the stroke feature weight:

[0195]

[0196] Wherein: t represents the current time step of reverse denoising, and T is the maximum time step;

[0197] Early stage (t≈0): slow weight decrease, prefer skeleton feature.

[0198] Late stage (t≈T): weight decreases rapidly, detail feature dominates.

[0199] Specifically, in the character repair task, the stroke order often has an important influence on the final visual effect. If the whole character image is directly denoised at the same time, it may cause the strokes to overlap or the order to be chaotic. Therefore, the present application introduces a "stroke order guided noise mask" in the original diffusion denoising process, so that the network only focuses on repairing the corresponding stroke area at different stages, and the stroke area that has been completed or is not in time is relatively stable.

[0200] In the reverse denoising process of the diffusion model, in order to realize more smooth feature fusion, the application adopts a dynamic weight scheduling mechanism, aiming to coordinate the contribution proportion of the skeleton feature (global structure) and the stroke feature (local detail), and ensure that the repair process conforms to the "structure priority-detail progressive" logic of Chinese character writing. For example, when repairing a damaged "horse" character, a noise mask is first generated, and the stroke area to be repaired is activated in stages according to the writing order. In the early time step (t is close to the total step T), the skeleton prior weight is high (such as λ(t)=0.9 λ ( t )=0.9), and the model preferentially restores the overall structure of the text; in the later time step (t is close to 0), the stroke prior weight gradually increases (such as λ(t)=0.1 λ ( t )=0.1), and the model focuses on refining the stroke edge.

[0201] Through iterative sampling, the noise is gradually removed and the strokes are repaired. For example, at the 100th reverse sampling, the model reconstructs the horizontal hook structure of the "horse" character according to the conditional vector; at the 20th sampling, the stroke details of the hook part are refined, and finally a complete character image is generated.

[0202] The application uses a noise mask to strengthen denoising in the spatial dimension by stroke stage, and smoothly transitions from skeleton prior to stroke prior in the time dimension. In the diffusion reverse sampling process, each stroke is activated and repaired in turn, and the overall structure is ensured in the early stage and the stroke details are polished in the later stage. This "dynamic stroke generation mechanism" not only makes the repair process more interpretable and controllable (can simulate the human writing order), but also effectively improves the level and detail accuracy of the repair effect, and achieves significant improvement in subjective visual evaluation and quantitative indicators.

[0203] In this way, in the character repair method based on the double-condition guided diffusion model according to the embodiment of the application, the global skeleton feature matrix obtained by modeling the character skeleton topology based on the preset complete character image accurately reconstructs the symmetric topology structure of the body; the stroke timing feature matrix obtained by modeling the timing features and morphological details based on the preset complete character image captures the starting and ending directions of the existing strokes of the left half; and then the cross-attention fusion ensures that the newly generated strokes of the right half are consistent with the style of the left half; at the same time, the dynamic generation mechanism is used to repair the tripod feet and tripod ears in stages, avoiding stroke sticking.

[0204] In the Chinese character test set, the structural error rate is significantly reduced, the broken stroke reconstruction success rate is significantly improved, and the calligraphy character stroke coherence is ensured. The experimental results show that the repair result is highly consistent with the real character distribution, and the edge definition is significantly improved. Compared with the original diffusion model, the reasoning speed is improved, the convergence is accelerated by the condition guidance, the data and computing power demand are reduced compared with the original diffusion model, and efficiency and resource optimization are realized.

[0205] The application is superior to the prior art in terms of structural accuracy, image quality, efficiency and resource optimization, and provides key technical support for cultural heritage protection and industrial intelligentization.

[0206] Exemplary electronic device

[0207] Below, an electronic device according to embodiments of the present application will be described with reference to Figure 2 The electronic device can be an electronic device integrated with a first imaging device, or a single device independent of the first imaging device, which can communicate with the first imaging device to receive input signals collected therefrom.

[0208] Figure 2 A block diagram of an electronic device according to embodiments of the present application is illustrated.

[0209] As Figure 2 shown, the electronic device 10 includes one or more processors 11 and a memory 12.

[0210] The processor 11 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 10 to perform desired functions.

[0211] The memory 12 can include one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage medium, and the processor 11 can run the program instructions to implement the diffusion model character repair method based on double condition guidance of various embodiments of the present application described above and / or other desired functions.

[0212] In one example, the electronic device 10 can also include an input device 13 and an output device 14, which are interconnected by a bus system and / or other forms of connection mechanism (not shown).

[0213] In addition, the input device 13 can include, for example, a keyboard, a mouse, and the like.

[0214] The output device 14 can output various information to the outside. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0215] Of course, in order to simplify, Figure 2 Only some of the components of the electronic device 10 related to the present application are shown in FIG. 1, and components such as a bus, an input / output interface, and the like are omitted. In addition to this, the electronic device 10 can include any other appropriate components according to the specific application.

[0216] Exemplary computer program product and computer readable storage medium.

[0217] In addition to the methods and devices described above, embodiments of the present application can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the diffusion model text restoration method based on double condition guidance according to various embodiments of the present application described in the above “Exemplary Method” section of the specification.

[0218] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server.

[0219] In addition, embodiments of the present application can also be a computer readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps of the moving object tracking method according to various embodiments of the present application described in the above “Exemplary Method” section of the specification.

[0220] The computer readable storage medium can be a combination of one or more computer readable media. The computer readable storage medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] The above description of the disclosed aspects merely exemplifies the general principles of the application. It is intended to be illustrative only since numerous modifications and variations of the present aspects are possible in light of the above teachings without departing from the spirit and intended scope of the application. It is therefore to be understood that changes can be made in the form, implementations, and details of the described implementations without departing from the intended spirit of the application.

[0222] The block diagrams of the devices, apparatuses, systems, etc. involved in the present application are only illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. shown in the block diagrams are required or implied. These devices, apparatuses, systems, etc. can be connected, arranged, configured, etc. in any manner as will be appreciated by those skilled in the art. The words comprising, including, having, containing, comprising, etc. are to be open-ended words that mean "including, but not limited to," and allow for other items to also be present. The words "or" and "and" as used herein are to be interpreted as logical or and logical and, respectively. The words "such as" are to be interpreted as "such as but not limited to."

[0223] It is also to be noted that the various components or steps in the apparatuses, devices, and methods of the present application can be combined, divided, re-arranged, omitted, eliminated, and / or re-arranged, depending on the specifics of the implementation. Such re-combinations, divisions, re-arrangements, omissions, eliminations, and / or re-arrangements are to be considered equivalents of the original sub-combinations, divisions, re-arrangements, omissions, eliminations, and / or re-arrangements.

[0224] The above description of disclosed aspects is meant to be illustrative only and not limiting. Many variations of the disclosure are possible as discernible by those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not to be limited to the aspects disclosed herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0225] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the application to the forms disclosed herein. Although various example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations, which fall within the scope of the application.

Claims

1. A text restoration method based on a dual-condition guided diffusion model, characterized in that, include: Obtain the first feature matrix and the second feature matrix; The first feature matrix is ​​a global skeleton feature matrix obtained by modeling the text skeleton topology based on a preset complete character image; The second feature matrix is ​​a stroke temporal feature matrix obtained by modeling based on temporal features and morphological details of a preset complete character image; The steps for obtaining the first feature matrix include: A preset complete character image is input into a skeleton graph encoder, which outputs a node feature matrix representing the global skeleton topology, resulting in a first feature matrix. The skeleton graph encoder is constructed based on a graph attention network. The implementation of the skeleton graph encoder includes: Extract a binarized skeleton image based on a preset complete character image; The binarized skeleton image is modeled as a graph structure G=(V,E); Wherein, the node set V represents the intersection of strokes, and the edge set E represents the connection relationship between strokes; The node neighborhood features are aggregated through a multi-layer graph attention network, and the node features are updated to the global skeleton feature matrix. The steps for obtaining the second feature matrix include: A preset complete character image is input into a stroke set encoder, which outputs a feature matrix encoding local stroke shapes and temporal dependencies, resulting in a second feature matrix. The stroke set encoder is constructed using a Transformer encoder. The implementation of the stroke set encoder includes: Based on a pre-defined complete character image, a stroke direction encoding sequence arranged in the writing order is extracted to obtain a normalized stroke direction vector sequence. Input a normalized sequence of stroke direction vectors and introduce sinusoidal position coding to preserve writing timing information; Self-attention weights are calculated using a multi-layer Transformer encoder, and the stroke temporal feature matrix is ​​output. The first feature matrix is ​​used as the query vector, and cross-attention fusion is performed with the key vector and value vector of the second feature matrix to generate the target condition vector; The target conditional vector is injected into the noise preset network of the diffusion model whose initial input is the character image to be repaired. The local features of the skip connections are dynamically fused with the target conditional vector through cross attention blocks to generate enhanced features. The enhanced features are linearly combined with the first feature matrix and the second feature matrix, and the restored character image is generated by weighting the skeleton prior and the stroke prior.

2. The text restoration method based on a dual-condition guided diffusion model as described in claim 1, characterized in that, The step of using the first feature matrix as a query vector and performing cross-attention fusion with the key vector and value vector of the second feature matrix to generate a target condition vector includes: The first feature matrix is ​​projected onto the query space, and the second feature matrix is ​​projected onto the key space and the value space, respectively. Calculate the similarity matrix between the query space and the key space, and normalize it using the Softmax function to obtain the attention weights; The target condition vector is obtained by weighted summation of the value vector based on the attention weights.

3. The text restoration method based on a dual-condition guided diffusion model as described in claim 1, characterized in that, The step of injecting the target conditional vector into a noisy preset network of a diffusion model whose initial input is the character image to be repaired, and dynamically fusing the local features of skip connections with the target conditional vector through cross-attention blocks to generate enhanced features, includes: At the skip connections of the noise pre-set network in the diffusion model, the first feature matrix is ​​used as a query and matched with the key vector and value vector of the second feature matrix; The context vector is calculated by scaling dot product attention, concatenated with a preset complete character image, and then fused by convolution to generate enhanced features.

4. The text restoration method based on a dual-condition guided diffusion model as described in claim 1, characterized in that, Before generating the repaired character image by weighting the skeleton prior and stroke prior, the method further includes: In the inverse denoising process of the diffusion model, a dynamic stroke generation mechanism is introduced to activate the stroke region to be repaired in stages through noise masking. The constraint formula for the noise mask is: The constraint formula for the noise mask is: （11）； in, The stroke regions that need to be repaired at the current stage are masked; noise reduction is enhanced in the masked regions by Hadamard product ⊙, while the original prediction is maintained in the non-masked regions. ,Right now , which represents the new noise prediction, which is the updated result obtained by the network after making stronger repairs to the stroke region in the current step; , represents the original noise prediction, that is, the noise estimate output by the model before any special processing of the stroke area; This indicates the noise prediction of the final output, which uses the new prediction value in the stroke area and retains the original prediction value in the non-stroke area. when When the time is specified, it indicates the stroke portion that needs repair, forcing the model to apply the newly predicted noise in the current step. Denoise the specified stroke area; when At the same time, the background parts other than the text retain the noise state of the previous time step to avoid interfering with areas that have been repaired or have not yet reached the repair stage; The phased activation is achieved by performing phased conditional weight scheduling through a dynamic weight formula. The dynamic weighting formula is as follows: ； In the initial stage, from t to T, there is a high-noise phase. Skeletal features dominate, and the model prioritizes restoring the global topological structure of Chinese characters to ensure the standardization of character shapes. Mid-term, t=T / 2, transition phase. The weight of skeleton features gradually decreases, while stroke features begin to participate in local shape optimization. Later, as t approaches 0, a low-noise phase begins. The model focuses on sharpening brushstrokes and reconstructing details of ink density to enhance visual realism.

5. An electronic device, comprising: processor; And a memory storing computer program instructions that, when executed by the processor, cause the processor to perform the text repair method based on a dual-condition guided diffusion model as described in any one of claims 1-4.

6. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the text repair method based on a dual-condition guided diffusion model as described in any one of claims 1-4.

Images