Image Registration Method and Device Based on Convolutional Network and Global Feature Extraction Network

Abstract

An image registration method and device based on a convolutional network and a global feature extraction network, comprising: extracting an image to be printed and colored from the original design drawing of a product as a fixed image, and extracting a corresponding image to be printed and colored from the finished product image of the product as a moving image; inputting the fixed image and the moving image into a deformable image registration network to generate a deformation field, and applying the deformation field to the moving image through a spatial transformation function to generate a corresponding deformed image, where the deformed image is used to replace the original design drawing as the printing and coloring reference image of the product; adjusting the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum. The present invention combines the advantages of a lightweight convolutional network and a Transformer, and realizes accurate image registration by processing local and global features in parallel.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to an image registration method and device based on a convolutional network and a global feature extraction network. Background Art

[0002] During the production process of zipper sliders, factories usually precisely emboss the finished zipper sliders according to the design drawings, and color the letters or logos on the surface of the zipper sliders through printing and coloring techniques. However, due to factors such as process errors, thermal expansion or deformation of materials, the letters or marks on the surface of the zipper sliders often deform during the compression molding process. This deformation not only affects the overall appearance of the zipper slider, reducing the aesthetic degree of the product, but also may cause printing color deviation during the coloring process, affecting the clarity and readability of the letters or logos, and even interfering with subsequent processing procedures.

[0003] To ensure the consistency and reliability of product quality and functions, it is urgent to apply high-precision calibration techniques and deformation repair algorithms to solve the displacement deviation problem of letters and other marks on the zipper slider during the coloring and printing process due to deformation, so as to ensure the accuracy and compliance of the final product.

[0004] Most current processing methods mainly rely on manual adjustment, which has many disadvantages such as time-consuming, low precision and poor consistency. Although some processes have begun to introduce neural networks or single deep learning models for deformation repair and correction, these methods often face limitations such as insufficient generalization ability and limited processing of complex deformation problems, and cannot meet the dual requirements of high precision and efficiency in the production process. Therefore, it is urgent to develop more intelligent and multi-level algorithms to improve the effect and application universality of deformation repair. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the problem of displacement deviation in coloring and printing caused by letter deformation during the forming process of zipper sliders in the prior art, and propose an image registration method and device based on a convolutional network and a global feature extraction network, which can correct the overall deformation of the zipper slider while ensuring the accuracy of letter details, thereby improving the quality and consistency of zipper sliders in production.

[0006] The present invention adopts the following technical solutions:

[0007] An image registration method based on a convolutional network and a global feature extraction network includes the following:

[0008] Extract the image to be printed and colored from the original design drawing of the product as the fixed image, and extract the corresponding image to be printed and colored from the finished product image of the product as the moving image;

[0009] Input the fixed image and the moving image into a deformable image registration network to generate a deformation field, and apply the deformation field to the fixed image through a spatial transformation function to generate a corresponding deformed image, which is used to replace the original design drawing as the printing and coloring reference image of the product;

[0010] Adjust the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum.

[0011] The deformable image registration network includes a U-shaped structure constructed by a lightweight convolutional network and a Transformer network, which includes a downsampling stage and an upsampling stage; the downsampling stage extracts local features and global features from the input image and performs splicing and fusion; the upsampling stage upsamples and restores the size of the feature map output by the downsampling stage, and splices and fuses the feature map output by the downsampling stage with the feature map in the upsampling through cross-layer connection, and then generates the deformation field.

[0012] In the downsampling stage, the convolutional module of the lightweight convolutional network is used to extract local features of the input image, the Transformer module of the Transformer network is used to extract global features, and a bridging block is used to splice and fuse the local features and the global features.

[0013] The convolutional module uses spatially separable convolution to extract local features, and the spatially separable convolution includes depth convolution and point convolution;

[0014] Assume the input feature map is , represents the height, represents the width, represents the number of channels of the input feature map; the depth convolution performs independent convolution on each channel, and the convolution kernel is , and the convolved feature map is:

[0015] ;

[0016] Among them, , * represents the convolution operation;

[0017] The point convolution fuses in the channel dimension through 1×1 convolution, and the convolution kernel , is the output number of channels of the local feature;

[0018] The fused local feature is:

[0019] ;

[0020] Among them 。

[0021] The Transformer module extracts global features through window partitioning and self-attention mechanism. Assuming the input feature map is , first perform window partitioning on the input feature map, divide the input feature map into N non-overlapping windows, and the size of each window is M×M; perform self-attention calculation on the features X within each window, including passing the feature X through linear transformations , , respectively to generate query Q, key K and value vector V:

[0022] ;

[0023] Among them, Q, K, V , represents the feature dimension;

[0024] Calculate the self-attention weight matrix A:

[0025] ;

[0026] Obtain the output through weighted summation by the attention mechanism:

[0027] ;

[0028] Among them, , is the global feature of each window.

[0029] Use a bridging block to splice and fuse the local feature and the global feature, specifically:

[0030] First, reshape the global feature into a four-dimensional form , and then splice the local feature with the transformed global feature , represents the splicing function, ; is the output channel number of the local feature, represents the channel number of the feature map output after the global feature branch is processed;

[0031] Then, use two 1×1×1 convolutional kernels W1, W2 to perform feature fusion on the spliced feature map.

[0032] The upsampling stage restores the size of the feature map output by the downsampling stage, and through cross-layer connection, splices and fuses the feature map output by the downsampling stage with the feature map after size restoration, and then processes to generate the deformation field.

[0033] The upsampling stage uses trilinear interpolation to restore the size of the feature map of the downsampling stage, specifically as follows: ;

[0034] where represents the trilinear interpolation function, represents the feature map of the downsampling stage, , where , and are the height, width, and depth after downsampling respectively, is the number of channels of downsampling, , represent the height, width, and depth of the feature map after restoration respectively, is the number of channels of upsampling;

[0035] The cross-layer connection splices the feature map of the downsampling stage with the feature map after size restoration to obtain the spliced feature map ; and fuses them through a convolution module using spatially separable convolution;

[0036] Finally, a 3D convolution module is used to generate the deformation field.

[0037] The formulas for calculating the similarity between the fixed image and the deformed image and adjusting the parameters of the deformable image registration network are as follows:

[0038] ;

[0039] where, is the optimal deformation field that maximizes the similarity; is to find an optimal deformation field parameter ; the deformed image = , represents the spatial transformation function; Φ is the parameter of the deformable registration network, represents the fixed image and the deformed image The similarity When the similarity is the largest, the deformable image registration network is in the optimal state, the deformation field has been correctly optimized, and the deformed image is the final registered image.

[0040] An image registration device based on a convolutional network and a global feature extraction network, comprising:

[0041] An image acquisition module, which extracts an image to be printed and colored from the original design drawing of the product as a fixed image, and extracts a corresponding image to be printed and colored from the finished product image of the product as a moving image;

[0042] An image registration module, which uses a deformable image registration network to generate a deformation field for the input fixed image and the moving image, and applies the deformation field to the moving image through a spatial transformation function to generate a corresponding deformed image, and the deformed image is used to replace the original design drawing as the printing and coloring reference image of the product;

[0043] A network adjustment module, which adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum.

[0044] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. Efficient feature extraction and fusion: The present invention combines Pyramid Vision Transformer and a lightweight convolutional module (LACB), making full use of the advantages of both. Pyramid Vision Transformer is good at capturing global dependencies in images, while LACB efficiently extracts local features. For the precise detection and correction of industrial zipper pieces, the fusion of such global and local features can more accurately identify subtle deformations and markings on the zipper pieces, thereby improving the accuracy of detection and correction.

[0046] 2. Reduction of computational amount and number of parameters: The present invention replaces the traditional 3D convolutional module with the lightweight convolutional module LACB, effectively reducing the computational amount and the number of parameters. This optimization is particularly suitable for large-scale and high-efficiency zipper piece detection in industrial environments, which can reduce the demand for computing resources while ensuring high-performance detection results.

[0047] 3. Flexible spatial transformation and precise registration: Through the deformation field and the spatial transformation function, the present invention can achieve precise deformation and registration of industrial zipper pieces. Optimizing the deformation field to maximize the similarity between the fixed image and the deformed image can effectively compensate for the deformations that may occur during the production process of the zipper pieces, thereby ensuring high-precision registration. This precise registration technology can significantly improve the accuracy of markings and patterns on zipper pieces and enhance product quality. Description of the Drawings

[0048] Figure 1This is the flowchart of the method of the present invention;

[0049] Figure 2 This is the flowchart of the application of the method of the present invention to the printing of zipper pieces;

[0050] Figure 3 This is the framework diagram of the deformable image registration network of the present invention;

[0051] Figure 4 This is the specific structural diagram of the deformable image registration network of the present invention.

[0052] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Specific embodiments

[0053] The present invention will be further described below through specific embodiments.

[0054] See Figures 1 to 4 , an image registration method based on a convolutional network and a global feature extraction network, including the following:

[0055] S1 Extract the image to be printed and colored as a fixed image from the original design drawing of the product, and extract the corresponding image to be printed and colored from the finished product image of the product as a moving image.

[0056] In this step, after obtaining the zipper piece dataset T, each pair of images in it is processed. One of them is the original design drawing of the zipper piece, and the image to be printed and colored (such as letters) is extracted as a fixed image, which is used to represent the standard design form before compression and molding; the other is the finished product image of the zipper piece product, and the corresponding image to be printed and colored is extracted as a moving image, which represents the actual product after compression and molding.

[0057] S2 Input the fixed image and the moving image into the deformable image registration network to generate a deformation field, and apply the deformation field to the fixed image through a spatial transformation function to generate a corresponding deformed image, and the deformed image is used to replace the original design drawing as the printing and coloring reference image of the product.

[0058] The variable image registration network of the present invention adopts the parallel image registration technology of a lightweight convolutional network (LOCV-Net) and a Transformer network (Pyramid Vision Transformer). The lightweight convolutional network is responsible for extracting the local details of the image to be printed and colored to ensure the effectiveness of deformation correction; the Transformer network captures global features and corrects the large-range deformation dependence. By fusing the features of the two, the efficient combination of local and global information is realized, so as to improve the correction effect of the overall deformation of the zipper piece while ensuring the accuracy of the details of the image to be printed and colored.

[0059] Among them, the deformable image registration network is constructed by a lightweight convolutional network and a Transformer network, that is, a U-shaped structure based on the LOCV-Net branch and the Pyramid Vision Transformer branch, which includes a downsampling stage and an upsampling stage. See Figure 4 .

[0060] The downsampling stage mainly extracts and fuses local features and global features from the input image. It extracts local features and global features from the input image and performs splicing and fusion. In practical applications, the convolutional module of the lightweight convolutional network is used in the downsampling stage to extract local features of the input image, the Transformer module of the Transformer network is used to extract global features, and a bridging block is used to splice and fuse local features and global features.

[0061] The downsampling stage includes two parallel branches. The local feature branch uses the convolutional module LACB (Lightweight Attention-based Convolution Block) of the lightweight convolutional network. This module reduces the number of parameters through depthwise separable convolution, thereby improving computational efficiency and effectively extracting local features of the image. At the same time, the global feature branch uses the Transformer module of the Transformer network to extract global features, that is, to capture global dependence information in the image and process long-range context relationships.

[0062] In local feature extraction, the convolutional module uses depthwise separable convolution to extract local features. Depthwise separable convolution includes depth convolution and point convolution. Assume the input feature map is , represents the height, represents the width, represents the number of channels of the input feature map. Depth convolution performs independent convolution on each channel, and the convolution kernel is , and the convolved feature map is:

[0063] ;

[0064] where, , * represents the convolution operation;

[0065] Point convolution fuses in the channel dimension through 1×1 convolution, and the convolution kernel , where is the output number of channels of the local feature. The fused local feature is:

[0066] ;

[0067] Among them 。

[0068] In global feature extraction, the Transformer module extracts global features through window partitioning and self-attention mechanism. Assuming the input feature map is , first perform window partitioning on the input feature map, divide the input feature map into N non-overlapping windows, and the size of each window is M×M; perform self-attention calculation on the feature X within each window, including passing the feature X through linear transformations respectively , , to generate query Q, key K, and value vector V:

[0069] ;

[0070] Among them, Q, K, V , represent the feature dimension;

[0071] Calculate the self-attention weight matrix A:

[0072] ;

[0073] Obtain the output through weighted summation by the attention mechanism:

[0074] ;

[0075] Among them, , is the global feature of each window.

[0076] In this embodiment, the feature maps extracted by the convolutional module LACB and the Transformer module are fused in the BridgingBlock. The BridgingBlock first concatenates these two feature maps through Concat, and then uses two parallel 1×1×1 convolutions for feature fusion to adjust the number of channels to meet the input requirements of the subsequent layers. In addition, the BridgingBlock is also responsible for converting the two-dimensional feature map (H×W×L,C) of the global feature into a four-dimensional feature map (H,W,L,C) for fusion with the feature map output by the LACB module, and finally converting the fused four-dimensional feature map into a two-dimensional form (H×W×L,C) to provide sufficient information for the upsampling stage.

[0077] Among them, the bridging block is used to splice and fuse the local feature and the global feature, specifically:

[0078] First, reshape the global feature into a four-dimensional form , and then splice the local feature with the transformed global feature , ; represents the splicing function, indicating the number of channels of the feature map output after processing by the global feature branch;

[0079] Next, two 1×1×1 convolutional kernels W1 and W2 are used to perform feature fusion on the spliced feature map.

[0080] The goal of the upsampling stage is to gradually restore the size of the feature map while fusing the feature information obtained in the downsampling stage. In the upsampling stage, the feature map output in the downsampling stage is upsampled and its size is restored, and the feature map output in the downsampling stage is spliced and fused with the feature map in the upsampling through cross-layer connection, and then a deformation field is generated.

[0081] Specifically, in the upsampling stage, trilinear interpolation is used to restore the size of the feature map in the downsampling stage as follows:

[0082] where, represents the trilinear interpolation function, represents the feature map in the downsampling stage, , where , and are the height, width, and depth after downsampling respectively, , represent the height, width, and depth of the feature map after restoration respectively; is the number of channels in the downsampling, is the number of channels in the upsampling.

[0083] Cross-layer connection splices the feature map in the downsampling stage with the feature map after size restoration to obtain the spliced feature map ; then it is fused through a convolutional module using spatially separable convolution:

[0084] Depth convolution:

[0085] , ;

[0086] where, is the depth convolution kernel, and k is the size of the convolution kernel.

[0087] Point convolution:

[0088] , ;

[0089] where, is the point convolution kernel, is the number of output channels, is the feature map after depth convolution, is the fused feature map.

[0090] Finally, a 3D convolution module is used to generate a displacement field. After upsampling through the decoder, a displacement field between the fixed image and the moving image is generated. The displacement field Φ is applied to the fixed image I using the spatial transformation function to generate the deformed image Î = .

[0091] S3 adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum. That is, by calculating the similarity between the fixed image I and the deformed image Î to evaluate the registration effect.

[0092] In this step, the formulas for calculating the similarity between the fixed image and the deformed image and adjusting the parameters of the deformable image registration network are as follows:

[0093] ;

[0094] where, is the best displacement field that maximizes the similarity; is to find an optimal displacement field parameter ; the deformed image Î = , represents the spatial transformation function; Φ is the parameter of the deformable registration network, represents the similarity between the fixed image I and the deformed image Î When the similarity is the largest, the deformable image registration network is in the optimal state, the displacement field has been correctly optimized, and the deformed image is the final registered image.

[0095] In the method of the present invention, first, the lightweight convolutional network LOCV-Net is used to extract the local features of the zipper piece letters or logos, which can effectively handle local detailed deformations, such as letter distortions, position offsets, etc. At the same time, the PyramidVision Transformer module is used to capture the global features and process the large-scale deformations generated during the compression process of the zipper piece. In order to effectively combine the local and global features, a bridging block is designed for feature fusion to ensure that the fused features meet the requirements of the next layer input. Through this method, when the model processes local and global deformations on the zipper piece, it can achieve precise correction, thereby ensuring the accuracy of the shape and position of the letters or logos during the subsequent coloring process, and greatly improving the quality and consistency of the product.

[0096] An image registration device based on a convolutional network and a global feature extraction network, which is used to execute the above-mentioned image registration method based on a convolutional network and a global feature extraction network, includes:

[0097] An image acquisition module, which extracts the image to be printed and colored from the original design drawing of the product as the fixed image, and extracts the corresponding image to be printed and colored from the finished product image of the product as the moving image. This image acquisition module is used to execute step S1 of the above method.

[0098] An image registration module, which uses a deformable image registration network to generate a deformation field for the input fixed image and moving image, and applies the deformation field to the moving image through a spatial transformation function to generate a corresponding deformed image. The deformed image is used to replace the original design drawing as the reference image for printing and coloring the product. This image registration module is used to execute step S2 of the above method.

[0099] A network adjustment module, which adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum. This network adjustment module is used to execute step S3 of the above method.

[0100] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0101] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described here can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on the network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0102] After considering the specification and practicing the invention disclosed here, those skilled in the art will easily think of other embodiments of the present disclosure. The present disclosure aims to cover any variations, uses, or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include the common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure.

[0103] The above are only specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantive modification of the present invention using this concept shall fall within the scope of infringement of the protection scope of the present invention.

Claims

1. An image registration method based on a convolutional network and a global feature extraction network, characterized in that It includes the following: Extract the image to be printed and colored as a fixed image from the original design drawing of the product, and extract the corresponding image to be printed and colored as a moving image from the finished product image of the product; Input the fixed image and the moving image into a deformable image registration network to generate a deformation field, and apply the deformation field to the fixed image through a spatial transformation function to generate a corresponding deformed image, which is used to replace the original design drawing as the printing and coloring reference image of the product; Adjust the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum; The deformable image registration network includes a U-shaped structure constructed by a lightweight convolutional network and a Transformer network, which includes a downsampling stage and an upsampling stage; in the downsampling stage, local features and global features of the input image are extracted and spliced and fused; in the upsampling stage, the feature map output by the downsampling stage is upsampled and the size is restored, and the feature map output by the downsampling stage is spliced and fused with the feature map in the upsampling through cross-layer connection, and then the deformation field is generated; In the upsampling stage, trilinear interpolation is used to restore the size of the feature map in the downsampling stage, specifically as follows: F up = TrilinearInterp(F down , (H up , W up , D up )); Among them, TrilinearInterp represents the trilinear interpolation function, F down represents the feature map in the downsampling stage, where H d , W d and D d are the height, width and depth after downsampling respectively, and C down is the number of channels of downsampling, H up , W up , D up represent the height, width and depth after the feature map is restored respectively, and C up is the number of channels of upsampling; The cross-layer connection connects the feature map in the downsampling stage with the feature map after size restoration to obtain a concatenated feature map through concatenation and fuses them through a convolutional module using depthwise separable convolution; Finally, a 3D convolutional module is used to generate the deformation field; The formulas for calculating the similarity between the fixed image and the deformed image and adjusting the parameters of the deformable image registration network are as follows: Φ * = argmax Φ Similarity(I, I); Among them, Φ * is the best deformation field that maximizes the similarity; argmax Φ is to find an optimal deformation field parameter Φ; the deformed image T(·) represents the spatial transformation function; Φ is the parameter of the deformable registration network, and Similarity(I, T(I, Φ)) represents the similarity between the fixed image I and the deformed image When the similarity is the largest, the deformable image registration network is in the optimal state, the deformation field has been correctly optimized, and the deformed image is the final registered image.

2. The image registration method based on a convolutional network and a global feature extraction network according to claim 1, wherein In the downsampling stage, the convolutional module of the lightweight convolutional network is used to extract the local features of the input image, the Transformer module of the Transformer network is used to extract the global features, and a bridging block is used to splice and fuse the local features and the global features.

3. The image registration method based on a convolutional network and a global feature extraction network according to claim 2, wherein The convolutional module uses spatially separable convolution to extract local features, and the spatially separable convolution includes depth convolution and point convolution; Assume the input feature map is H up represents the height, W up represents the width, C in represents the number of channels of the input feature map; the depth convolution performs independent convolution on each channel, and the convolution kernel is K d ∈R k×k and the convolved feature map F d is: F d = X * K d ; Among them, * represents the convolution operation; The point convolution performs fusion in the channel dimension through 1×1 convolution, and the convolution kernel where C out is the output channel number of the local feature, and C down is the channel number of downsampling; The fused local feature F LACB is as follows: F LACB = F d * K p ; Among them 4. The image registration method based on a convolutional network and a global feature extraction network according to claim 2, wherein The Transformer module extracts global features through window partitioning and self-attention mechanism. Assume the input feature map is First, perform window partitioning on the input feature map, dividing the input feature map into N non-overlapping windows, each window with a size of M×M; perform self-attention calculation on the features X within each window, including passing the features X through linear transformations W q , W k , W v to generate query Q, key K, and value vector V: Q = XW q , K = XW k , V = XW v ; Among them d represents the feature dimension; Calculate the self-attention weight matrix A: Obtain the output by weighted summation through the attention mechanism: F Swin = AV; Among them, is the global feature of each window.

5. The image registration method based on a convolutional network and a global feature extraction network according to claim 2, characterized in that, A bridging block is used to splice and fuse the local features and the global features. Specifically: First, reshape the global feature F Swin to a four-dimensional form Then, concatenate the local feature F TACB with the reshaped global feature Concat represents the concatenation function, and the concatenated feature map is C out is the number of output channels of the local feature, and C global represents the number of channels of the feature map output after processing by the global feature branch; Next, two 1×1×1 convolutional kernels W1 and W2 are used to perform feature fusion on the spliced feature map.

6. The image registration method based on a convolutional network and a global feature extraction network according to claim 2, wherein The upsampling stage restores the size of the feature map output by the downsampling stage, and splices and fuses the feature map output by the downsampling stage with the size-restored feature map through cross-layer connection, and then processes and generates the deformation field.

7. An image registration device based on a convolutional network and a global feature extraction network, characterized in that, An image registration method based on a convolutional network and a global feature extraction network for implementing the method recited in claim 1 includes: An image acquisition module that extracts the image to be printed and colored as a fixed image from the original design drawing of the product, and extracts the corresponding image to be printed and colored as a moving image from the finished product image of the product; An image registration module that uses a deformable image registration network to generate a deformation field for the input fixed image and moving image, and applies the deformation field to the moving image through a spatial transformation function to generate a corresponding deformed image, which is used to replace the original design drawing as the printing and coloring reference image of the product; The network adjustment module adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum.

Images