An image processing method and system based on geometric consistency constraints

Abstract

The application relates to the technical field of computer vision and image processing, and particularly discloses an image processing method and system based on geometric consistency constraints, which comprises the following steps: constructing a structure-texture collaborative generator, the structure-texture collaborative generator performs collaborative processing on an image based on texture feature data flow and geometric feature data flow; performing cyclic consistency training on the structure-texture collaborative generator based on geometric constraints to obtain a trained structure-texture collaborative generator; and inputting a degraded initial image into the trained structure-texture collaborative generator to obtain a target image. Through the construction of the structure-texture collaborative generator and the introduction of geometric constraints of an edge space in cyclic consistency training, the accuracy of geometric structure recovery and consistency maintenance in the style migration process are ensured while the physical defect repair of a degraded scanning sheet is realized, error accumulation is avoided, and thus the digitalization quality of high-precision images such as industrial drawings and scientific charts is significantly improved.

Description

Technical Field

[0001] This disclosure generally relates to the fields of computer vision and image processing technology, and specifically to an image processing method and system based on geometric consistency constraints. Background Technology

[0002] In the field of modern computer vision, image style transfer and image inpainting are two crucial research directions. Image inpainting aims to recover high-fidelity original details from damaged or degraded inputs, while style transfer strives to achieve cross-domain transformation of visual appearance while preserving the semantic meaning of the content. Although these two tasks have made breakthrough progress in processing natural images such as landscapes and portraits with the iteration of deep learning technology, existing technologies still face serious challenges when processing high-precision images containing high-frequency geometric information (such as industrial blueprints, scientific charts, and precision design drawings).

[0003] In practical applications, it is often necessary to simultaneously address image texture restoration and style transfer. Related techniques typically employ a step-by-step strategy, performing image restoration first, followed by style transfer. While this pipeline is feasible for insensitive natural images, the cumulative effect of errors is particularly severe in high-precision geometric image processing. If geometric accuracy is not perfectly restored during the image restoration stage, residual minor structural defects, such as discontinuities in lines, will be treated as image content features during the style transfer stage, further amplified or distorted, potentially leading to complete breaks or artifacts. This step-by-step operation severs the intrinsic connection between restoration and transfer tasks, lacking a unified geometric consistency constraint mechanism. Therefore, how to collaboratively achieve high-quality texture restoration and style transfer within a unified framework, while ensuring the integrity of high-precision geometric structures, is a critical challenge that urgently needs to be addressed in this field. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the existing technology, it is desirable to provide an image processing method and system based on geometric consistency constraints. By constructing a structure-texture co-generator and introducing geometric constraints of the edge space in the cyclic consistency training, the method can not only repair the physical defects of degraded scanned parts, but also ensure the accurate restoration of geometric structure and the maintenance of consistency in the style transfer process, avoid error accumulation, and thus significantly improve the digital quality of high-precision images such as industrial drawings and scientific charts.

[0005] In a first aspect, embodiments of this application provide an image processing method based on geometric consistency constraints, including:

[0006] A structure-texture co-generator is constructed, which performs co-processing on images based on texture feature data streams and geometric feature data streams.

[0007] The structure-texture co-generator is subjected to cyclic consistency training based on geometric constraints to obtain the trained structure-texture co-generator;

[0008] The degraded initial image is input into the trained structure-texture co-generator to obtain the target image.

[0009] In some embodiments, the structure-texture co-generator includes a texture inpainting module, an explicit structure injection module, and a fusion feature encoding module. The step of inputting the degraded initial image into the trained structure-texture co-generator to obtain the target image includes:

[0010] The initial image is input to the texture restoration module and the explicit structure injection module, respectively;

[0011] The texture restoration module uses a shallow convolutional layer to extract initial features from the initial image to obtain an initial feature map; then it uses a window-based self-attention mechanism unit to extract texture features from the initial feature map to obtain the texture features corresponding to the initial image.

[0012] The explicit structure injection module uses an edge extraction operator to extract edge features from the initial image to obtain an edge map corresponding to the initial image; a lightweight convolutional layer is used to perform feature transformation on the edge map to obtain the geometric features corresponding to the initial image.

[0013] The fusion feature encoding module generates the target image based on the texture features and the geometric features.

[0014] In some embodiments, the geometrically constrained cyclic consistency training trains the structure-texture co-generator to obtain a trained structure-texture co-generator, including:

[0015] Based on the structure-texture co-generator, a structure-texture loop generator corresponding to the structure-texture co-generator is constructed, which is used to restore the output image of the structure-texture co-generator to the input image;

[0016] Based on the input image and output image of the structure-texture co-generator, and the output image of the structure-texture loop generator, the structure-texture co-generator is jointly optimized using a multi-scale discriminator.

[0017] In some embodiments, the joint optimization of the structure-texture co-generator based on a multi-scale discriminator, using the input image, output image, and output image of the structure-texture recurring generator, includes:

[0018] During the training process, the ground truth image corresponding to the input image is obtained;

[0019] Based on the ground truth image and the output image of the structure-texture co-generator, determine the first structural consistency loss and style-aware loss;

[0020] The second structural consistency loss is determined based on the input image of the structure-texture co-generator and the output image of the structure-texture cyclic generator.

[0021] The structure-texture co-generator is jointly optimized based on the first structural consistency loss, the second structural consistency loss, and the style-aware loss.

[0022] In some embodiments, the step of performing cyclic consistency training on the structure-texture co-generator based on geometric constraints to obtain the trained structure-texture co-generator further includes:

[0023] For each optimization iteration, the quality of the structure-texture co-generator is evaluated based on conventional metrics and structural sensitivity metrics.

[0024] When the evaluation results meet the preset conditions, the trained structure-texture co-generator is obtained.

[0025] In some embodiments, the quality evaluation of the structure-texture co-generator based on the structure sensitivity index includes:

[0026] For any training iteration, the structural overlap and endpoint continuity penalty between the output image and the ground truth image of the structure-texture loop generator are obtained respectively.

[0027] Based on the structural overlap and the endpoint continuity penalty, the structure-texture co-generator is determined according to the structure sensitivity index.

[0028] Secondly, embodiments of this application provide an image processing system based on geometric consistency constraints, including:

[0029] A construction module is used to build a structure-texture co-generator, which performs co-processing of images based on texture feature data streams and geometric feature data streams.

[0030] The training module is used to perform cyclic consistency training on the structure-texture co-generator based on geometric constraints to obtain the trained structure-texture co-generator.

[0031] The application module is used to input the degraded initial image into the trained structure-texture co-generator to obtain the target image.

[0032] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in embodiments of this application.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in embodiments of this application.

[0034] Fifthly, embodiments of this application provide a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the method described in embodiments of this application.

[0035] The image processing method and system based on geometric consistency constraints provided in this application constructs a structure-texture co-generator, which performs collaborative image processing based on texture feature data streams and geometric feature data streams. The structure-texture co-generator undergoes cyclic consistency training based on geometric constraints to obtain a trained structure-texture co-generator. A degraded initial image is input into the trained structure-texture co-generator to obtain a target image. When processing high-precision images such as industrial blueprints, scientific charts, and precision design drawings, the generated vectorized style image is not only visually clear but also retains strict geometric regularities, meeting the application requirements of engineering-level digitization. Moreover, thanks to its powerful geometric preservation capabilities, it significantly reduces manual intervention steps such as "line splicing" and "text / symbol repair" commonly found in automated digitization processes, improving overall operational efficiency.

[0036] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0037] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0038] Figure 1 A schematic flowchart of an image processing method based on geometric consistency constraints according to an embodiment of this application is shown;

[0039] Figure 2 A schematic diagram illustrating the principle of an image processing method based on geometric consistency constraints provided in an embodiment of this application is shown.

[0040] Figure 3 A schematic diagram of the explicit structure injection module provided in one embodiment of this application is shown;

[0041] Figure 4 A schematic diagram illustrating the principle of structural consistency loss provided in an embodiment of this application is shown;

[0042] Figure 5 A schematic diagram illustrating the principle of style perception loss provided in an embodiment of this application is shown;

[0043] Figure 6 A schematic diagram of the structure of an image processing system based on geometric consistency constraints according to an embodiment of this application is shown;

[0044] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an electronic device or server according to embodiments of this application is shown. Detailed Implementation

[0045] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] In the field of modern computer vision, image inpainting and style transfer are two core problems. Image inpainting aims to restore the original details of damaged or degraded images, while style transfer involves transforming the style of an image from one domain to another. Although significant progress has been made in both tasks in natural image processing, existing technologies face major challenges when processing high-precision geometric images such as industrial blueprints and scientific charts, particularly in preserving the geometric structure and restoring details within the image.

[0048] Image style transfer technology has undergone a leapfrog development from traditional methods based on physical models to those based on deep generative models. Early nonparametric methods mainly relied on texture synthesis, making it difficult to extract high-level semantics. With the introduction of convolutional neural network-based methods that utilized feature separation to achieve artistic style transfer, the paradigm of deep style transfer was established. Subsequently, researchers introduced maximum mean difference, Markov random fields, and fast iterative optimization models to improve the quality and speed of generated textures. The emergence of generative adversarial networks further propelled the development of this field, especially the proposal of unsupervised models such as CycleGAN and DiscoGAN, which broke through the limitations of pairwise data. In recent years, diffusion models and cross-modal techniques, represented by Stable Diffusion and CLIP, have significantly improved the fidelity of generated images and the controllability of text.

[0049] However, most of the methods mentioned above are designed for natural images or works of art, focusing on matching statistical texture characteristics and often ignoring the inherent topological structure of the image, resulting in a common problem of lacking geometric consistency. When applied to industrial drawings or scientific charts, these methods are prone to destroying the fine geometric elements in the image.

[0050] Topological disruption: High-frequency geometric structures such as small curves, symbols, and scales are often misidentified as texture noise and smoothed or distorted, causing deformation or breakage of the topological structure of the drawing.

[0051] Loss of accuracy: In scenarios such as industrial design where geometric consistency is strictly required, any geometric distortion caused by style transfer can lead to the loss of critical information, resulting in design errors or production mistakes. While existing methods have achieved decoupling between style and content, they cannot effectively model and protect high-frequency geometric features explicitly.

[0052] Image inpainting aims to address degradation issues such as noise, blurring, and compression artifacts. Its technological paradigm has evolved from traditional filtering algorithms to complex network architectures based on deep learning. Current mainstream methods include MIRNet based on multi-scale encoders and decoders, IPT incorporating the global perception capabilities of Transformers, and algorithms that utilize generative adversarial networks to simulate realistic noise. Recently, diffusion models and cross-modal large models combined with CLIP have also been introduced into inpainting tasks, demonstrating powerful detail generation capabilities. Furthermore, specialized models for specific domains, such as remote sensing mapping, medical imaging, and cultural relic restoration, are emerging, improving restoration results in specific scenarios by incorporating prior knowledge or multi-task joint optimization.

[0053] While these techniques excel at restoring local texture self-similarity in natural scenes, they still face significant bottlenecks when processing geometrically complex engineering images:

[0054] High-frequency detail blurring: Industrial blueprints contain a large number of precise lines, arcs and characters, and existing methods tend to produce smooth results, resulting in blurring of high-frequency geometric edges.

[0055] Geometric semantic breaks: The lack of explicit constraints on geometric features during image restoration makes it impossible to guarantee the continuity of lines and the integrity of symbols when filling in damaged areas. Even minute pixel-level distortions or curve breaks are unacceptable errors in digital engineering drawings, severely impacting their accuracy and usability.

[0056] Based on this, this application proposes an image processing method and system based on geometric consistency constraints. By constructing a structure-texture co-generator and introducing geometric constraints of the edge space in the cyclic consistency training, it can achieve the repair of physical defects in degraded scanned parts, while ensuring the accurate restoration of geometric structure and the maintenance of consistency in the style transfer process, avoiding error accumulation, thereby significantly improving the digital quality of high-precision images such as industrial drawings and scientific charts.

[0057] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation instruction steps as shown in the following embodiments or drawings, the method may include more or fewer operation instruction steps based on conventional or non-creative effort. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0058] Please refer to Figure 1 , Figure 1 A schematic flowchart of an image processing method based on geometric consistency constraints according to an embodiment of this application is shown. Figure 1 As shown, the method includes:

[0059] Step 101: Construct a structure-texture co-generator. The structure-texture co-generator performs co-processing on the image based on the texture feature data stream and the geometric feature data stream.

[0060] It should be noted that, in order to collaboratively optimize the background denoising and structure preservation tasks in the image processing of degraded images, this application designs a generator with a dual-stream parallel architecture, aiming to unify the decoupling of physical degradation noise and semantic content within the feature space. Specifically, within the same network framework, it simultaneously achieves the removal of physical degradation (noise, stains, creases) and the preservation of geometric structures (curves, symbols, scales), avoiding the smoothing of high-frequency geometric information during the restoration process in traditional single-task models.

[0061] In one feasible embodiment, the structure-texture co-generator includes a texture restoration module, an explicit structure injection module, and a fusion feature encoding module.

[0062] In other words, in this embodiment, a dual-stream parallel architecture is adopted to decompose the image into a "texture stream" (responsible for background denoising and semantic restoration) and a "structure stream" (responsible for explicit injection of edges and geometric skeletons), and fuse them in the decoding stage. This allows the network to both fill large-area defects using long-distance texture context and constrain edge sharpness using geometric priors.

[0063] The texture inpainting module extracts deep texture features from the image to repair non-geometric defects (such as stains, blur, and background noise) caused by scanning degradation, while maintaining an understanding of the global context. Given the importance of long-range pixel dependencies for image restoration, this application employs a window-based self-attention mechanism as the core coding unit. Specifically, the initial image is input into the texture inpainting module within the structure-texture co-generator. Shallow convolutional layers are used to extract initial features from the initial image, resulting in an initial feature map. This initial feature map is then input into stacked Swing Transformer blocks, utilizing its moving window self-attention mechanism to calculate the correlation between pixels within a non-overlapping window, thereby obtaining the texture features corresponding to the initial image. These texture features primarily include the background texture after noise removal, color distribution, and potential image context information.

[0064] It should be understood that the texture inpainting module uses the Swing Transformer as its core encoding unit. Its Moving Window Self-Attention (W-MSA) mechanism can efficiently compute long-range pixel dependencies within non-overlapping windows, thus enabling logically consistent filling using distantly similar background textures when repairing large areas of dirt. Compared to convolutional networks, the Transformer's global receptive field gives it a greater advantage in handling large-area degradation.

[0065] The explicit structure injection module actively extracts and injects geometric structure information (edges, curve skeletons) into the image during the encoding stage. This compensates for the smoothing defects of the Transformer texture flow in high-frequency geometric details, ensuring that the sharpness of curves and intersections is not lost during generation. Specifically, the initial image is input into the explicit structure injection module in the structure-texture co-generator, such as... Figure 2 As shown, the explicit structure injection module first extracts edge features from the initial image using the Sobel operator to obtain the edge map corresponding to the initial image. Then, a set of lightweight convolutional layers is used to transform the edge map to obtain the geometric features corresponding to the initial image. The combined features obtained by the explicit structure injection module serve as a combination guide, explicitly injected into multiple layers of the texture inpainting module to compensate for the detail smoothing problem that may occur when Transformer processes extremely high-frequency edges. It should be understood that this explicit structure injection is equivalent to providing the network with a "geometric attention map," forcing the network to prioritize the preservation of edge responses during decoding, rather than relying on the network to learn (which it often fails to) high-frequency preservation. This effectively solves the problem of high-frequency edge response attenuation and curve detail loss caused by the self-attention smoothing effect of the Transformer model. This results in sharp curve edges and clear intersections in the generated image, and the geometric skeleton is always forcibly constrained during style transfer, fundamentally avoiding geometric distortion.

[0066] The fusion feature encoding module integrates texture features rich in background context with structural features of a precise geometric skeleton, and reconstructs an output image that both repairs physical defects and preserves geometric integrity through a decoder. Specifically, channels are concatenated to form hybrid features, and the decoder employs a symmetrical upsampling structure to progressively restore spatial resolution and refine the output using residual connections. The decoding process is guided by the fusion features in two ways: texture features provide color and texture consistency, while structural features provide edge sharpness constraints. It should be understood that by using the fusion feature encoding module, texture restoration and geometry can be kept unified in the same generator, avoiding the error accumulation caused by two-stage separation, resulting in an output image that is visually sharp and clear while maintaining a complete and unbroken geometric structure.

[0067] It should also be understood that, compared with the traditional "repair first, then transfer" cascaded scheme, the structure-texture co-generator trained using the embodiments of this application eliminates the sense of separation between tasks, enabling the repaired structure to directly guide style generation, and avoiding the technical risk that the initial repair error will be nonlinearly amplified in the subsequent transfer stage.

[0068] Step 102: Perform cyclic consistency training based on geometric constraints on the structure-texture co-generator to obtain the trained structure-texture co-generator.

[0069] It should be noted that this application constructs a bidirectional mapping mechanism of "dual generators and dual discriminators." It not only utilizes a structure-texture co-generator to convert degraded images into vector images, but also uses a structure-texture loop generator to restore the vector images generated by the structure-texture co-generator back to the original degraded images, forming closed-loop supervision. Simultaneously, dual discriminators are configured for adversarial training to improve the perceptual realism of the generated images.

[0070] In one feasible embodiment, the structure-texture co-generator is trained using geometrically constrained cyclic consistency training to obtain the trained structure-texture co-generator, including: constructing a structure-texture cyclic generator corresponding to the structure-texture co-generator based on the structure-texture co-generator, used to restore the output image of the structure-texture co-generator to the input image; and jointly optimizing the structure-texture co-generator based on a multi-scale discriminator according to the input image, output image, and output image of the structure-texture co-generator.

[0071] Furthermore, based on the input and output images of the structure-texture co-generator and the output image of the structure-texture recurrent generator, the structure-texture co-generator is jointly optimized using a multi-scale discriminator. This includes: during training, obtaining the ground truth image corresponding to the input image; determining the first structural consistency loss and style-aware loss based on the ground truth image and the output image of the structure-texture co-generator; determining the second structural consistency loss based on the input image of the lost structure-texture co-generator and the output image of the structure-texture recurrent generator; and jointly optimizing the structure-texture co-generator based on the first structural consistency loss, the second structural consistency loss, and the style-aware loss.

[0072] Specifically, such as Figure 3 As shown, the structure-texture co-generator processes the input image through a series of operations by an encoder and decoder to obtain the generated image. The structure-texture loop generator processes the generated image output by the structure-texture co-generator through a series of operations by an encoder and decoder to obtain the loop reconstructed image. On the input side of the structure-texture co-generator, a second structural consistency loss is determined based on the input image and the loop reconstructed image output by the structure-texture loop generator. On the output side of the structure-texture co-generator, a first structural consistency loss and a style perception loss are determined based on the generated image output by the structure-texture co-generator and the ground truth image corresponding to the input image. The ground truth image can be the target style image corresponding to the input image obtained through manual restoration or other methods, or it can be a target style image that does not have a one-to-one correspondence with the input image.

[0073] It should be noted that, in order to force the model to prioritize maintaining the consistency of the geometric skeleton during style transfer and iterative reconstruction, rather than blindly reconstructing background pixels, thereby avoiding the retention of background blemishes as "structure" and preventing curve breakage during style transfer, this application proposes a structural consistency loss. The core idea of ​​structural consistency loss is the loss of edge feature differences, used to avoid geometric distortion during style transfer. Figure 4 As shown, taking the second structural consistency loss as an example, the input image of the structure-texture co-generator and the output image (cyclic reconstruction image) of the structure-texture loop generator are used to extract edge features using the edge extraction method used by the explicit structure injection module, thereby obtaining the second structural consistency loss.

[0074] For example, the structural consistency loss can be expressed as:

[0075]

[0076] in, For structural consistency loss, This is the first structural loss. This is the second structural loss. Edge features of the ground truth image The edge features of the output image of the structure-texture co-generator. The edge features of the input image are used as input to the structure-texture co-generator. The edge features of the output image are used by the structure-texture loop generator. This is the cycle coefficient.

[0077] It should be understood that the construction of structural consistency loss solves the problem of traditional cyclic consistency loss forcibly preserving background artifacts in degraded image scenes; and solves the problems of curve breakage and topological distortion in style transfer.

[0078] Style-aware loss is used to ensure that the generated image maintains consistency with the target vector domain in terms of high-level semantics and visual style, avoiding style shifts or blurring caused by pixel-level loss alone. For example... Figure 5 As shown, style-aware loss uses trained convolutional layers to extract feature maps and calculates the Euclidean distance between the generated image and the target style image, thereby achieving accurate transfer of style texture.

[0079] For example, the style-aware loss function can be expressed as:

[0080]

[0081] in, For style perception loss, The number of feature extraction layers. The extracted features of the j-th layer of the ground truth image, Extract the features of the j-th layer of the output image for the structure-texture co-generator.

[0082] In a preferred embodiment, the structure-texture loop generator is trained end-to-end under the joint supervision of robust loss, repair loss, structural consistency loss, and style-aware loss, and the total loss function can be expressed as:

[0083]

[0084] in, For the total loss, To combat the losses, The standard inpainting loss for image denoising, For structural consistency loss, For style perception loss, , These are the weighting coefficients.

[0085] It should be understood that, in the embodiments of this application, the four losses are mutually restraining— The driver generates realistic images. Constrained physical denoising Forced geometry preservation, This guides style transfer, ultimately enabling the structure-texture co-generator to learn a mapping that removes degradation without destroying geometric features.

[0086] In one feasible embodiment, the structure-texture co-generator is subjected to cyclic consistency training based on geometric constraints to obtain the trained structure-texture co-generator. The method further includes: for each optimization iteration, the structure-texture co-generator is evaluated based on conventional indicators and structural sensitivity indicators; when the evaluation results meet the preset conditions, the trained structure-texture co-generator is obtained.

[0087] In other words, in order to objectively evaluate and guide the structure-texture loop generator to achieve high-quality structure recovery during training, this application introduces a joint evaluation mechanism of conventional indicators and structure-sensitive indicators.

[0088] Common metrics include, but are not limited to, PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), LPIPS (Learned Perceptual Image Patch Similarity), FID (Frechet Inception Distance), KID, and ArtFID, which are used to measure pixel fidelity, perceptual similarity, and distribution consistency.

[0089] The structural sensitivity index is a quantitative indicator that is sensitive to geometric topological fractures. It is used for model selection and training monitoring, and makes up for the shortcomings of conventional indicators such as PSNR in reflecting line integrity.

[0090] In one feasible embodiment, the quality evaluation of the structure-texture co-generator based on the structure sensitivity index includes: for any training iteration, obtaining the structural overlap and endpoint continuity penalty between the output image and the ground truth image of the structure-texture recurrent generator; and determining the structure sensitivity index corresponding to the structure-texture co-generator based on the structural overlap and endpoint continuity penalty.

[0091] In other words, the structural sensitivity index skeletonizes both the generated and ground truth images and calculates two dimensions: curve alignment (spatial overlap accuracy) and endpoint continuity penalty (topological integrity). The structural sensitivity index is the product of these two; a high score is achieved only when both accurate localization and unbroken continuity are simultaneously satisfied.

[0092] For example, the structural sensitivity index can be expressed as:

[0093]

[0094] in, As a structural sensitivity index, To output the skeleton data of the image, The skeleton data is for the ground truth image. For structural overlap, Penalty for endpoint continuity.

[0095] in:

[0096]

[0097]

[0098] P represents the precision after morphological skeletonization, and represents the recall rate. This is the number of endpoints (i.e., the number of free ends of skeleton lines) within the 8-connected domains of the output image. λ is the number of endpoints in the 8-connected region of the true image (i.e., the number of free ends of the skeleton lines), and λ is the weighting coefficient.

[0099] It should be understood that the embodiments of this application effectively solve the problem that conventional indicators are not sensitive to curve breaks by constructing structural sensitivity indicators, and provide objective values ​​for geometric integrity.

[0100] It should also be noted that, in this embodiment, model training and parameter setting are performed end-to-end, with full parameter fine-tuning of the network on a large-scale dataset. The AdamW optimizer is used, with the initial learning rate set to... The cosine annealing process is used to gradually decay the image to zero. During training, the input image is cropped to a fixed resolution (512×512) for batch training until the model converges.

[0101] Step 103: Input the degraded initial image into the trained structure-texture co-generator to obtain the target image.

[0102] In other words, after the structure-texture co-generator is trained and converged and evaluated by conventional metrics and structural sensitivity metrics, it can be directly applied to the processing of degraded images. That is, the initial degraded image is directly input into the trained structure-texture co-generator to obtain the target image.

[0103] In summary, the image processing method based on geometric consistency constraints provided in this application constructs a structure-texture co-generator, which performs collaborative image processing based on texture feature data streams and geometric feature data streams. The structure-texture co-generator undergoes cyclic consistency training based on geometric constraints to obtain a trained structure-texture co-generator. The degraded initial image is input into the trained structure-texture co-generator to obtain the target image. When processing high-precision images such as industrial blueprints, scientific charts, and precision design drawings, the generated vectorized style map is not only visually clear but also retains strict geometric regularities, meeting the application requirements of engineering-level digitization. Moreover, thanks to its powerful geometric preservation capabilities, it significantly reduces manual intervention steps such as "line patching" and "text / symbol repair" commonly found in automated digitization processes, improving overall operational efficiency.

[0104] It should be noted that although the operation of the method of the present invention is described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed in order to achieve the desired result.

[0105] Figure 6 A schematic diagram of the structure of an image processing system based on geometric consistency constraints provided in an embodiment of this application is shown.

[0106] like Figure 6 As shown, the image processing system 10 based on geometric consistency constraints includes:

[0107] Module 11 is used to construct a structure-texture co-generator, which performs co-processing of images based on texture feature data streams and geometric feature data streams;

[0108] Training module 12 is used to perform cyclic consistency training on the structure-texture co-generator based on geometric constraints to obtain the trained structure-texture co-generator;

[0109] Application module 13 is used to input the degraded initial image into the trained structure-texture co-generator to obtain the target image.

[0110] In some embodiments, the structure-texture co-generator includes a texture restoration module, an explicit structure injection module, and a fusion feature encoding module. Application module 13 is specifically used for:

[0111] The initial image is input to the texture restoration module and the explicit structure injection module, respectively;

[0112] The texture restoration module uses a shallow convolutional layer to extract initial features from the initial image to obtain an initial feature map; then it uses a window-based self-attention mechanism unit to extract texture features from the initial feature map to obtain the texture features corresponding to the initial image.

[0113] The explicit structure injection module uses an edge extraction operator to extract edge features from the initial image to obtain an edge map corresponding to the initial image; a lightweight convolutional layer is used to perform feature transformation on the edge map to obtain the geometric features corresponding to the initial image.

[0114] The fusion feature encoding module generates the target image based on the texture features and the geometric features.

[0115] In some embodiments, the training module 12 is specifically used for:

[0116] Based on the structure-texture co-generator, a structure-texture loop generator corresponding to the structure-texture co-generator is constructed, which is used to restore the output image of the structure-texture co-generator to the input image;

[0117] Based on the input image and output image of the structure-texture co-generator, and the output image of the structure-texture loop generator, the structure-texture co-generator is jointly optimized using a multi-scale discriminator.

[0118] In some embodiments, the training module 12 is specifically used for:

[0119] During the training process, the ground truth image corresponding to the input image is obtained;

[0120] Based on the ground truth image and the output image of the structure-texture co-generator, determine the first structural consistency loss and style-aware loss;

[0121] The second structural consistency loss is determined based on the input image of the structure-texture co-generator and the output image of the structure-texture cyclic generator.

[0122] The structure-texture co-generator is jointly optimized based on the first structural consistency loss, the second structural consistency loss, and the style-aware loss.

[0123] In some embodiments, the training module 12 is specifically used for:

[0124] For each optimization iteration, the quality of the structure-texture co-generator is evaluated based on conventional metrics and structural sensitivity metrics.

[0125] When the evaluation results meet the preset conditions, the trained structure-texture co-generator is obtained.

[0126] In some embodiments, the training module 12 is specifically used for:

[0127] For any training iteration, the structural overlap and endpoint continuity penalty between the output image and the ground truth image of the structure-texture loop generator are obtained respectively.

[0128] Based on the structural overlap and the endpoint continuity penalty, the structure-texture co-generator is determined according to the structure sensitivity index.

[0129] It should be understood that the modules or modules described in the image processing system 10 based on geometric consistency constraints are related to the reference modules. Figure 1 The steps in the described method correspond accordingly. Therefore, the operations and features described above for the method are also applicable to the image processing system 10 based on geometric consistency constraints and its included modules, and will not be repeated here. The image processing system 10 based on geometric consistency constraints can be pre-implemented in the browser or other security applications of an electronic device, or it can be loaded into the browser or its security applications of an electronic device through download or other means. The corresponding modules in the image processing system 10 based on geometric consistency constraints can cooperate with modules in the electronic device to implement the solutions of the embodiments of this application.

[0130] The division of modules or units mentioned in the detailed description above is not mandatory. In fact, according to the embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0131] The following is for reference. Figure 7 , Figure 7 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of this application is shown.

[0132] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. RAM 703 also stores various programs and data required for the system's operating instructions. The CPU 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0133] The following components are connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 710 as needed so that computer programs read from it can be installed into storage section 708 as needed.

[0134] Specifically, according to embodiments of this application, the flowchart above refers to... Figure 2 The described process can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program contains program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined in the system of this application.

[0135] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operational instructions of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two connected blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified functions or operational instructions, or using a combination of dedicated hardware and computer instructions.

[0137] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a construction module, a training module, and an application module. The names of these units or modules do not necessarily limit the unit or module itself. For example, a construction module can also be described as "building a structure-texture co-generator, which performs co-processing of images based on texture feature data streams and geometric feature data streams."

[0138] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments, or may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the image processing method based on geometric consistency constraints described in this application.

[0139] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. An image processing method based on geometric consistency constraints, characterized in that, include: A structure-texture co-generator is constructed, which performs co-processing on images based on texture feature data streams and geometric feature data streams. The structure-texture co-generator is subjected to cyclic consistency training based on geometric constraints to obtain the trained structure-texture co-generator; The degraded initial image is input into the trained structure-texture co-generator to obtain the target image.

2. The image processing method based on geometric consistency constraints according to claim 1, characterized in that, The structure-texture co-generator includes a texture inpainting module, an explicit structure injection module, and a fusion feature encoding module. The process of inputting the degraded initial image into the trained structure-texture co-generator to obtain the target image includes: The initial image is input to the texture restoration module and the explicit structure injection module, respectively; The texture restoration module uses a shallow convolutional layer to extract initial features from the initial image to obtain an initial feature map; then it uses a window-based self-attention mechanism unit to extract texture features from the initial feature map to obtain the texture features corresponding to the initial image. The explicit structure injection module uses an edge extraction operator to extract edge features from the initial image to obtain an edge map corresponding to the initial image; a lightweight convolutional layer is used to perform feature transformation on the edge map to obtain the geometric features corresponding to the initial image. The fusion feature encoding module generates the target image based on the texture features and the geometric features.

3. The image processing method based on geometric consistency constraints according to claim 1, characterized in that, The geometrically constrained cyclic consistency training is used to train the structure-texture co-generator to obtain the trained structure-texture co-generator, including: Based on the structure-texture co-generator, a structure-texture loop generator corresponding to the structure-texture co-generator is constructed, which is used to restore the output image of the structure-texture co-generator to the input image; Based on the input image and output image of the structure-texture co-generator, and the output image of the structure-texture loop generator, the structure-texture co-generator is jointly optimized using a multi-scale discriminator.

4. The image processing method based on geometric consistency constraints according to claim 3, characterized in that, The step of jointly optimizing the structure-texture co-generator based on the input image, output image, and output image of the structure-texture loop generator, using a multi-scale discriminator, includes: During the training process, the ground truth image corresponding to the input image is obtained; Based on the ground truth image and the output image of the structure-texture co-generator, determine the first structural consistency loss and style-aware loss; The second structural consistency loss is determined based on the input image of the structure-texture co-generator and the output image of the structure-texture cyclic generator. The structure-texture co-generator is jointly optimized based on the first structural consistency loss, the second structural consistency loss, and the style-aware loss.

5. The image processing method based on geometric consistency constraints according to claim 1, characterized in that, The step of performing cyclic consistency training based on geometric constraints on the structure-texture co-generator to obtain the trained structure-texture co-generator further includes: For each optimization iteration, the quality of the structure-texture co-generator is evaluated based on conventional metrics and structural sensitivity metrics. When the evaluation results meet the preset conditions, the trained structure-texture co-generator is obtained.

6. The image processing method based on geometric consistency constraints according to claim 5, characterized in that, The quality evaluation of the structure-texture co-generator based on the structure sensitivity index includes: For any training iteration, the structural overlap and endpoint continuity penalty between the output image and the ground truth image of the structure-texture loop generator are obtained respectively. Based on the structural overlap and the endpoint continuity penalty, the structure-texture co-generator is determined according to the structure sensitivity index.

7. An image processing system based on geometric consistency constraints, characterized in that, include: A construction module is used to build a structure-texture co-generator, which performs co-processing of images based on texture feature data streams and geometric feature data streams. The training module is used to perform cyclic consistency training on the structure-texture co-generator based on geometric constraints to obtain the trained structure-texture co-generator. The application module is used to input the degraded initial image into the trained structure-texture co-generator to obtain the target image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image processing method based on geometric consistency constraints as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the image processing method based on geometric consistency constraints as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the image processing method based on geometric consistency constraints as described in any one of claims 1-6.

Images