Image registration method and device based on convolutional network and global feature extraction network

Abstract

The invention discloses an image registration method and device based on a convolutional network and a global feature extraction network, and the method comprises the steps: extracting a to-be-printed and colored image from an original design drawing of a product as a fixed image, and extracting a corresponding to-be-printed and colored image from a finished product image of the product as a moving image; the fixed image and the moving image are input into a deformable image registration network to generate a deformation field, the deformation field is applied to the moving image through a spatial transformation function, a corresponding deformed image is generated, and the deformed image is used for replacing the original design drawing to serve as a printing coloring reference image of the product; and adjusting parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the fixed image and the deformed image reaches the maximum. According to the method, the advantages of the lightweight convolutional network and the Transform are combined, and accurate image registration is realized through parallel processing of local and global features.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image registration method and device based on a convolutional network and a global feature extraction network. Background Art

[0002] During the production process of zipper sheets, factories usually accurately emboss the finished zipper sheets according to the design drawings, and use printing and coloring technology to color the letters or logos on the surface of the zipper sheets. However, due to factors such as process errors, thermal expansion or deformation of materials, the letters or logos on the surface of the zipper sheets often deform during the compression molding process. This deformation not only affects the overall appearance of the zipper sheet and reduces the aesthetics of the product, but may also cause printing color deviations during the coloring process, affecting the clarity and readability of the letters or logos, and even interfering with subsequent processing procedures.

[0003] In order to ensure the consistency and reliability of product quality and functions, it is urgent to apply high-precision calibration technology and deformation repair algorithms to solve the displacement deviation problem caused by deformation of letters and other logos on zipper pieces during the color printing process, so as to ensure the accuracy and compliance of the final product.

[0004] Most current processing methods rely mainly on manual adjustments, which have many disadvantages such as time-consuming, low precision and poor consistency. Although some processes have begun to introduce neural networks or single deep learning models for deformation repair and correction, these methods often face limitations such as insufficient generalization ability and limited handling of complex deformation problems, and cannot meet the dual needs of high precision and efficiency in the production process. Therefore, it is urgent to develop more intelligent and multi-level algorithms to improve the effect and wide application of deformation repair. Summary of the invention

[0005] The main purpose of the present invention is to overcome the problem of printing color displacement deviation caused by letter deformation during the zipper piece forming process in the prior art, and propose an image registration method and device based on a convolutional network and a global feature extraction network, which can correct the overall deformation of the zipper piece while ensuring the accuracy of the letter details, thereby improving the quality and consistency of the zipper piece in production.

[0006] The present invention adopts the following technical solution:

[0007] An image registration method based on a convolutional network and a global feature extraction network, comprising the following:

[0008] Extracting an image to be printed and colored from the original design drawing of the product as a fixed image, and extracting a corresponding image to be printed and colored from the finished product image of the product as a moving image;

[0009] Inputting the fixed image and the moving image into a deformable image registration network to generate a deformation field, applying the deformation field to the fixed image through a spatial transformation function to generate a corresponding deformed image, wherein the deformed image is used to replace the original design drawing as a reference image for printing and coloring of the product;

[0010] The parameters of the deformable image registration network are adjusted by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches a maximum.

[0011] The deformable image registration network includes a U-shaped structure constructed by a lightweight convolutional network and a Transformer network, which includes a downsampling stage and an upsampling stage; the downsampling stage extracts local features and global features from the input image and performs splicing and fusion; the upsampling stage upsamples and resizes the feature map output by the downsampling stage, and splices and fuses the feature map output by the downsampling stage with the feature map in the upsampling through cross-layer connections to regenerate the deformation field.

[0012] The downsampling stage uses the convolution module of the lightweight convolutional network to extract local features of the input image, uses the Transformer module of the Transformer network to extract global features, and uses a bridging block to splice and fuse the local features and the global features.

[0013] The convolution module uses spatially separable convolution to extract local features, and the spatially separable convolution includes depth convolution and point convolution;

[0014] Assume that the input feature map is , Indicates height, Indicates width, Represents the number of channels of the input feature map; the depth convolution performs independent convolution on each channel, and the convolution kernel is , the feature map after convolution for:

[0015] ;

[0016] in, , * represents the convolution operation;

[0017] The point convolution is fused in the channel dimension through 1×1 convolution, and the convolution kernel , is the number of output channels of local features;

[0018] The local features after fusion for:

[0019] ;

[0020] in .

[0021] The Transformer module extracts global features through window partitioning and self-attention mechanism. Assuming that the input feature map is First, the input feature map is divided into N non-overlapping windows, each with a size of M×M; the feature X in each window is self-attention calculated, including linearly transforming the feature X , , Generate query Q, key K and value vector V:

[0022] ;

[0023] Where Q, K, V , Represents feature dimension;

[0024] Calculate the self-attention weight matrix A:

[0025] ;

[0026] The output is obtained by weighted summation through the attention mechanism:

[0027] ;

[0028] in, , is the global feature of each window.

[0029] The local features and the global features are spliced ​​and merged using a bridging block, specifically:

[0030] First, the global feature Perform reshape transformation and convert to four-dimensional form , and then the local features Concatenate with the converted global features , represents the concatenation function, ; is the number of output channels of local features, Indicates the number of channels of the feature map output by the global feature branch after processing;

[0031] Next, two 1×1×1 convolution kernels W1 and W2 are used to fuse the concatenated feature maps.

[0032] The upsampling stage restores the size of the feature map output by the downsampling stage, and concatenates and fuses the feature map output by the downsampling stage with the feature map after size restoration through cross-layer connection, and then processes and generates the deformation field.

[0033] The up-sampling stage uses trilinear interpolation to restore the size of the feature map in the down-sampling stage, as follows: ;

[0034] in represents the trilinear interpolation function, Represents the feature map of the downsampling stage, ,in , and are the height, width and depth after downsampling, is the number of downsampled channels, , Represent the height, width and depth of the restored feature map respectively, is the number of upsampled channels;

[0035] The cross-layer connection connects the feature map of the downsampling stage Feature map after size restoration Perform splicing to obtain the spliced ​​feature map ; Fusion is performed via convolutional modules using spatially separable convolutions;

[0036] Finally, a 3D convolution module is used to generate the deformation field.

[0037] The formula for calculating the similarity between the fixed image and the deformed image and adjusting the parameters of the deformable image registration network is as follows:

[0038] ;

[0039] in, is the optimal deformation field that maximizes the similarity; is to find an optimal deformation field parameter ; The deformed image = , represents the spatial transformation function; Φ is the parameter of the deformable registration network, Indicates the fixed image With the deformed image Similarity between When the similarity is maximum, the deformable image registration network is in an optimal state, the deformation field has been correctly optimized, and the deformed image is the final registered image.

[0040] An image registration device based on a convolutional network and a global feature extraction network, comprising:

[0041] An image acquisition module extracts an image to be printed and colored from the original design drawing of the product as a fixed image, and extracts a corresponding image to be printed and colored from the finished product image of the product as a moving image;

[0042] An image registration module, which uses a deformable image registration network to generate a deformation field from the input fixed image and the moving image, applies the deformation field to the moving image through a spatial transformation function, and generates a corresponding deformed image, wherein the deformed image is used to replace the original design drawing as a reference image for printing and coloring of the product;

[0043] The network adjustment module adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches a maximum.

[0044] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. Efficient feature extraction and fusion: This paper combines the Pyramid Vision Transformer and the Lightweight Convolutional Module (LACB), making full use of the advantages of both. The Pyramid Vision Transformer is good at capturing global dependencies in images, while the LACB efficiently extracts local features. For the precise detection and correction of industrial zippers, this fusion of global and local features can more accurately identify subtle deformations and marks on the zipper, thereby improving the accuracy of detection and correction.

[0046] 2. Reduce the amount of calculation and parameters: This invention adopts the lightweight convolution module LACB to replace the traditional 3D convolution module, effectively reducing the amount of calculation and parameters. This optimization is particularly suitable for large-scale and efficient zipper detection in industrial environments, which can reduce the demand for computing resources while ensuring high-performance detection results.

[0047] 3. Flexible spatial transformation and precise registration: Through the deformation field and spatial transformation function, the present invention can achieve precise deformation and registration of industrial zipper pieces. Optimizing the deformation field to maximize the similarity between the fixed image and the deformed image can effectively compensate for the deformation that may occur in the production process of the zipper piece, thereby ensuring high precision of registration. This precise registration technology can significantly improve the accuracy of the marking and pattern of the zipper piece and improve product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1is a flow chart of the method of the present invention;

[0049] Figure 2 A flow chart of the method of the present invention applied to zipper sheet printing;

[0050] Figure 3 This is a deformable image registration network framework diagram of the present invention;

[0051] Figure 4 This is a specific structural diagram of the deformable image registration network of the present invention.

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0053] The present invention is further described below through specific implementation modes.

[0054] See also Figures 1 to 4 , an image registration method based on a convolutional network and a global feature extraction network, comprising the following:

[0055] S1 extracts an image to be printed and colored from the original design drawing of the product as a fixed image, and extracts a corresponding image to be printed and colored from the finished product image of the product as a moving image.

[0056] In this step, after obtaining the zipper sheet data set T, each pair of images therein is processed, one of which is the original design drawing of the zipper sheet, and the image to be printed and colored (such as letters) is extracted as a fixed image to represent the standard design form before compression molding; the other is the finished product image of the zipper sheet product, and the corresponding image to be printed and colored is extracted as a moving image to represent the actual product after compression molding.

[0057] S2 inputs the fixed image and the moving image into the deformable image registration network to generate a deformation field, applies the deformation field to the fixed image through the spatial transformation function, and generates the corresponding deformed image. The deformed image is used to replace the original design drawing as the printing and coloring reference image of the product.

[0058] The variable image registration network of the present invention adopts the parallel image registration technology of lightweight convolutional network (LOCV-Net) and Transformer network (Pyramid Vision Transformer). The lightweight convolutional network is responsible for extracting the local details of the image to be printed and colored to ensure the effectiveness of deformation correction; the Transformer network captures the global features and corrects the large-scale deformation dependency. By integrating the features of the two, the efficient combination of local and global information is achieved, thereby ensuring the accuracy of the details of the image to be printed and colored while improving the correction effect of the overall deformation of the zipper.

[0059] Among them, the deformable image registration network includes a lightweight convolutional network and a Transformer network construction, that is, a U-shaped structure built based on the LOCV-Net branch and the Pyramid Vision Transformer branch, which includes a downsampling stage and an upsampling stage, see Figure 4 .

[0060] The downsampling stage mainly extracts and fuses local features and global features from the input image. It extracts local features and global features from the input image and performs splicing and fusion. In practical applications, the downsampling stage uses the convolution module of the lightweight convolutional network to extract local features of the input image, the Transformer module of the Transformer network to extract global features, and the bridging block to splice and fuse local features and global features.

[0061] The downsampling stage includes two parallel branches. The local feature branch uses the convolution module LACB (Lightweight Attention-based Convolution Block) of the lightweight convolutional network, which reduces the number of parameters through spatially separable convolution, thereby improving computational efficiency and effectively extracting local features of the image. At the same time, the global feature branch uses the Transformer module of the Transformer network to extract global features, that is, to capture global dependency information in the image and process long-distance contextual relationships.

[0062] In local feature extraction, the convolution module uses spatially separable convolution to extract local features. Spatially separable convolution includes depthwise convolution and pointwise convolution. Assume that the input feature map is , Indicates height, Indicates width, Indicates the number of channels of the input feature map. The depth convolution performs independent convolution on each channel, and the convolution kernel is , the feature map after convolution for:

[0063] ;

[0064] in, , * represents the convolution operation;

[0065] Point convolution is fused in the channel dimension through 1×1 convolution, and the convolution kernel ,in is the number of output channels of local features. The fused local features for:

[0066] ;

[0067] in .

[0068] In global feature extraction, the Transformer module extracts global features through window division and self-attention mechanism. Assuming that the input feature map is First, the input feature map is divided into N non-overlapping windows, each with a size of M×M; the feature X in each window is self-attention calculated, including the feature X is linearly transformed , , Generate query Q, key K and value vector V:

[0069] ;

[0070] Where Q, K, V , Represents feature dimension;

[0071] Calculate the self-attention weight matrix A:

[0072] ;

[0073] The output is obtained by weighted summation through the attention mechanism:

[0074] ;

[0075] in, , is the global feature of each window.

[0076] In this embodiment, the feature maps extracted by the convolution module LACB and the Transformer module are fused in the bridging block. The bridging block first concatenates the two feature maps through Concat, and then uses two parallel 1×1×1 convolutions to fuse the features, adjusting the number of channels to meet the input requirements of the subsequent layers. In addition, the bridging block is also responsible for converting the two-dimensional feature map (H×W×L, C) of the global feature into a four-dimensional feature map (H, W, L, C) for fusion with the feature map output by the LACB module, and finally converting the fused four-dimensional feature map into a two-dimensional form (H×W×L, C) to provide sufficient information for the upsampling stage.

[0077] Among them, the bridging block is used to splice and fuse local features and global features, specifically:

[0078] First, the global features Perform reshape transformation and convert to four-dimensional form , and then the local features Concatenate with the converted global features , ; represents the concatenation function, Indicates the number of channels of the feature map output by the global feature branch after processing;

[0079] Next, two 1×1×1 convolution kernels W1 and W2 are used to fuse the concatenated feature maps.

[0080] The goal of the upsampling stage is to gradually restore the size of the feature map and fuse the feature information obtained in the downsampling stage. The upsampling stage upsamples and restores the size of the feature map output in the downsampling stage, and concatenates and fuses the feature map output in the downsampling stage with the feature map in the upsampling stage through cross-layer connections to regenerate the shape field.

[0081] Specifically, the upsampling stage uses trilinear interpolation to restore the size of the feature map in the downsampling stage, as follows:

[0082] in, represents the trilinear interpolation function, Represents the feature map of the downsampling stage, ,in , and are the height, width and depth after downsampling, , Represent the height, width and depth of the restored feature map respectively; is the number of downsampled channels, is the number of upsampled channels.

[0083] The cross-layer connection downsamples the feature map Feature map after size restoration Perform splicing to obtain the spliced ​​feature map ; Then fuse them through a convolution module using spatially separable convolution:

[0084] Depthwise Convolution:

[0085] , ;

[0086] in, is the depth of the convolution kernel, and k is the size of the convolution kernel.

[0087] Point convolution:

[0088] , ;

[0089] in, is the point convolution kernel, is the number of output channels, is the feature map after deep convolution, It is the fused feature map.

[0090] Finally, the 3D convolution module is used to generate the displacement field. After upsampling through the decoder, the displacement field between the fixed image and the moving image is generated. The spatial transformation function is used to apply the displacement field Φ to the fixed image I to generate the deformed image Î= .

[0091] S3 adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum. That is, by calculating the similarity between the fixed image I and the deformed image Î To evaluate the registration effect.

[0092] In this step, the formula for calculating the similarity between the fixed image and the deformed image and adjusting the parameters of the deformable image registration network is as follows:

[0093] ;

[0094] in, is the optimal deformation field that maximizes the similarity; is to find an optimal deformation field parameter ; The deformed image Î= , represents the spatial transformation function; Φ is the parameter of the deformable registration network, Represents the similarity between the fixed image I and the deformed image Î When the similarity is maximum, the deformable image registration network is in the optimal state, the deformation field has been correctly optimized, and the deformed image is the final registered image.

[0095] The method of the present invention first uses a lightweight convolutional network LOCV-Net to extract local features of the letters or logos on the zipper sheet, which can effectively deal with local detail deformations, such as distortion of letters, position shift, etc. At the same time, the PyramidVision Transformer module is used to capture global features and process the large-scale deformations generated during the compression of the zipper sheet. In order to achieve an effective combination of local and global features, a bridge block is designed for feature fusion to ensure that the fused features meet the requirements of the next layer of input. Through this method, the model can achieve accurate correction when processing local and global deformations on the zipper sheet, thereby ensuring the accuracy of the shape and position of the letters or logos in the subsequent coloring process, greatly improving the quality and consistency of the product.

[0096] An image registration device based on a convolutional network and a global feature extraction network, used to execute the above-mentioned image registration method based on a convolutional network and a global feature extraction network, comprising:

[0097] The image acquisition module extracts the image to be printed and colored from the original design drawing of the product as a fixed image, and extracts the corresponding image to be printed and colored from the finished product image of the product as a moving image. The image acquisition module is used to execute step S1 of the above method.

[0098] The image registration module uses a deformable image registration network to generate a deformation field from the input fixed image and the moving image, applies the deformation field to the moving image through a spatial transformation function, and generates a corresponding deformed image, which is used to replace the original design drawing as a reference image for printing and coloring of the product. The image registration module is used to perform step S2 of the above method.

[0099] The network adjustment module adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches the maximum. The network adjustment module is used to perform step S3 of the above method.

[0100] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0101] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0102] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0103] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. An image registration method based on a convolutional network and a global feature extraction network, characterized in that: These include: Extracting an image to be printed and colored from the original design drawing of the product as a fixed image, and extracting a corresponding image to be printed and colored from the finished product image of the product as a moving image; Inputting the fixed image and the moving image into a deformable image registration network to generate a deformation field, applying the deformation field to the fixed image through a spatial transformation function to generate a corresponding deformed image, wherein the deformed image is used to replace the original design drawing as a reference image for printing and coloring of the product; The parameters of the deformable image registration network are adjusted by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches a maximum.

2. The image registration method based on a convolutional network and a global feature extraction network as claimed in claim 1, characterized in that: The deformable image registration network includes a U-shaped structure constructed by a lightweight convolutional network and a Transformer network, which includes a downsampling stage and an upsampling stage; the downsampling stage extracts local features and global features from the input image and performs splicing and fusion; the upsampling stage upsamples and resizes the feature map output by the downsampling stage, and splices and fuses the feature map output by the downsampling stage with the feature map in the upsampling through cross-layer connections to regenerate the deformation field.

3. The image registration method based on convolutional network and global feature extraction network as claimed in claim 2, characterized in that: The downsampling stage uses the convolution module of the lightweight convolutional network to extract local features of the input image, uses the Transformer module of the Transformer network to extract global features, and uses a bridging block to splice and fuse the local features and the global features.

4. The image registration method based on a convolutional network and a global feature extraction network as claimed in claim 3, characterized in that: The convolution module uses spatially separable convolution to extract local features, and the spatially separable convolution includes depth convolution and point convolution; Assume that the input feature map is , Indicates height, Indicates width, Represents the number of channels of the input feature map; the depth convolution performs independent convolution on each channel, and the convolution kernel is , the feature map after convolution for: ； in, , * represents the convolution operation; The point convolution is fused in the channel dimension through 1×1 convolution, and the convolution kernel , is the number of output channels of local features, is the number of downsampled channels; The local features after fusion for: ； in .

5. The image registration method based on convolutional network and global feature extraction network as claimed in claim 3, characterized in that: The Transformer module extracts global features through window partitioning and self-attention mechanism. Assuming that the input feature map is First, the input feature map is divided into N non-overlapping windows, each with a size of M×M; the feature X in each window is self-attention calculated, including linearly transforming the feature X , , Generate query Q, key K and value vector V: ； Where Q, K, V , Represents feature dimension; Calculate the self-attention weight matrix A: ； The output is obtained by weighted summation through the attention mechanism: ； in, , is the global feature of each window.

6. The image registration method based on convolutional network and global feature extraction network as claimed in claim 3, characterized in that: The local features and the global features are spliced ​​and merged using a bridging block, specifically: First, the global feature Perform reshape transformation and convert to four-dimensional form , and then the local features Concatenate with the converted global features , represents the concatenation function, ; is the number of output channels of local features, Indicates the number of channels of the feature map output by the global feature branch after processing; Next, two 1×1×1 convolution kernels W1 and W2 are used to fuse the concatenated feature maps.

7. The image registration method based on convolutional network and global feature extraction network as claimed in claim 2, characterized in that: The upsampling stage restores the size of the feature map output by the downsampling stage, and concatenates and fuses the feature map output by the downsampling stage with the feature map after size restoration through cross-layer connection, and then processes and generates the deformation field.

8. The image registration method based on convolutional network and global feature extraction network as claimed in claim 7, characterized in that: The up-sampling stage uses trilinear interpolation to restore the size of the feature map in the down-sampling stage, as follows: ; in represents the trilinear interpolation function, Represents the feature map of the downsampling stage, ,in , and are the height, width and depth after downsampling, is the number of downsampled channels, , Represent the height, width and depth of the restored feature map respectively, is the number of upsampled channels; The cross-layer connection connects the feature map of the downsampling stage Feature map after size restoration Perform splicing to obtain the spliced ​​feature map ; Fusion is performed via convolutional modules using spatially separable convolutions; Finally, a 3D convolution module is used to generate the deformation field.

9. The image registration method based on convolutional network and global feature extraction network as claimed in claim 7, characterized in that: The formula for calculating the similarity between the fixed image and the deformed image and adjusting the parameters of the deformable image registration network is as follows: ； in, is the optimal deformation field that maximizes the similarity; is to find an optimal deformation field parameter ; The deformed image = , represents the spatial transformation function; Φ is the parameter of the deformable registration network, Indicates the fixed image With the deformed image The similarity between When the similarity is maximum, the deformable image registration network is in an optimal state, the deformation field has been correctly optimized, and the deformed image is the final registered image.

10. An image registration device based on a convolutional network and a global feature extraction network, characterized in that: include: An image acquisition module extracts an image to be printed and colored from the original design drawing of the product as a fixed image, and extracts a corresponding image to be printed and colored from the finished product image of the product as a moving image; An image registration module, which uses a deformable image registration network to generate a deformation field from the input fixed image and the moving image, applies the deformation field to the moving image through a spatial transformation function, and generates a corresponding deformed image, wherein the deformed image is used to replace the original design drawing as a reference image for printing and coloring of the product; The network adjustment module adjusts the parameters of the deformable image registration network by calculating the similarity between the fixed image and the deformed image until the similarity between the two reaches a maximum.

Images