Printed matter image registration method based on convolution cross attention mechanism

Abstract

The application discloses a printed matter image registration method based on a convolution cross attention mechanism, first constructs a deep learning registration network, including a convolution cross attention mechanism and a deep homography estimation registration network based on up-sampling; then inputs a reference printed matter image and a printed matter image to be registered into the deep learning registration network to obtain a registered printed matter image; finally, the network parameters are optimized by calculating a loss function between the registered printed matter image and the reference printed matter image, and a printed matter image with higher precision is output. The application completes the printed matter image registration task, provides guarantee for defect detection of the printed matter in the later period, and improves the defect detection efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep neural network and image analysis, and particularly relates to a printed matter image registration method based on a convolution cross attention mechanism. BACKGROUND

[0002] The printing industry is an important industrial support of China's national economy. In daily life, books, magazines, newspapers, gift packaging boxes, business cards and the like belong to the category of printed matter. Printed matter is closely related to people's lives. However, in the production process of printed matter, due to external factors or internal equipment reasons, some defects such as ink flying, missing printing, color difference and cutting deviation will inevitably occur. In the defect detection technology of printed matter, the most primary task is registration, that is, aligning two printed matter images in geometry to ensure that the reference printed matter image and the printed matter image to be registered are aligned in spatial position, and then performing defect detection. The quality of registration determines the accuracy of defect detection, and has important research significance and practical value.

[0003] The continuous development of deep learning technology provides a new idea for the printed matter image registration method. The image registration method based on deep learning is not limited to feature extraction, but can also estimate the geometric transformation between images through a neural network to align. The unsupervised deep homography estimation model does not depend on real labels, and can optimize the training network by calculating the similarity measure of the registration image and the reference image. Not only can the features be learned, but also the homography can be estimated, and the image registration effect is good for images with large displacement and illumination change. The image registration of printed matter can effectively align two printed matter images in spatial position, and provide guarantee for the later defect detection of printed matter and improve the defect detection efficiency. SUMMARY

[0004] The purpose of the present application is to provide a printed matter image registration method based on a convolution cross attention mechanism, to complete the printed matter image registration task, to provide guarantee for the later defect detection of printed matter, and to improve the defect detection efficiency.

[0005] The technical solution adopted by the present application is a printed matter image registration method based on a convolution cross attention mechanism, which is implemented according to the following steps:

[0006] Step 1, constructing a deep learning registration network including a convolution cross attention mechanism and a deep homography estimation registration network based on upsampling;

[0007] Step 2, inputting the patch (p B ) of the reference printed matter image and the patch (p A ) of the printed matter image to be registered into the deep learning registration network to predict p Afour corner points of the reference printed image p B four corner points of the reference printed image p 4pt , the transformation matrix H' is obtained by direct linear transformation through DLT;

[0008] Step 3, performing spatial transformation on the transformation matrix H' and the printed image A to be registered to obtain a registered printed image;

[0009] Step 4, optimizing the network parameters by calculating the loss function between the registered printed image and the reference printed image, and outputting a printed image with higher precision.

[0010] The application is also characterized in that,

[0011] The convolution cross-attention mechanism in step 1 is implemented according to the following steps:

[0012] Step 1.1, input two given shape tensors X1 and X2, X1, X2 ∈ R H×W×C , H represents the height of the input feature map, W represents the width of the input feature map, C represents the channel number of the input feature map, and the size of X1 and X2 is 64*64*32;

[0013] Step 1.2, to ensure that the image processing contains translation invariance, the existing relative position encoding is extended to two dimensions, the relative position embedding of cross-attention is expanded to two dimensions, that is, the width information and height information of the pixel i=(i x , i y ) are embedded, and two-dimensional relative cross-attention is realized. The attention degree of pixel j=(j x , j y ) is calculated as formula (1):

[0014]

[0015] Where l i，j represents the attention degree of pixel i=(i x , i y ) to pixel j=(j x , j y ), represents the transpose of the pixel i query vector, represents the depth of the key k, k j is the key vector of pixel j, and represent the relative width j x -i x and the relative height j y -i y ;

[0016] Step 1.3, the output of two-dimensional single-head cross-attention is formula (2):

[0017]

[0018] where O h represents the output of two-dimensional single-head cross-attention, softmax(·) represents normalization, W Q represents the weight of the query, W k represents the weight of the key, and W v represents the weight of the value. represents the logical matrix of the relative position of width and height, X1 represents the tensor form of the feature Figure 1 , and X2 represents the tensor form of the feature Figure 2 . represents the depth of the key k.

[0019] Step 1.4, multi-head attention is spliced by single-head attention, as shown in formula (3):

[0020] MHA(X)=Concat[O1,...,O Nh ]W O (3)

[0021] where MHA(X) represents a multi-head attention tensor with a shape of (H, W, d v ), Concat[·] represents splicing, O1,..., O Nh represent single-head attention, and W represents a weight vector.

[0022] Step 1.5, the convolution and multi-head cross-attention feature map are mapped and connected to obtain the convolution cross-attention, which can be written as formula (4):

[0023] AAConv(X)=Concat[Conv(X),MHA(X)] (4)

[0024] where AAConv(X) represents convolution cross-attention, Concat[·] represents splicing, Conv(X) represents convolution, and MHA(X) represents a multi-head attention tensor with a shape of (H, W, d v ).

[0025] Step 1.6, the convolution cross-attention is batch normalized to obtain a feature map X'1 that integrates X2 features, and the size of X'1 is 64x64x32. Replace X1 with X'1.

[0026] The registration network based on the depth homography estimation of the upsampling in step 1 is implemented according to the following steps:

[0027] Step a, concatenate the feature map tensor X1 and the feature map tensor X2 obtained in step 1 to obtain a single feature map with a size of 64x64x64;

[0028] Step b, perform a transformation operation on the feature map with a size of 64x64x64;

[0029] Step c, perform a transformation operation on the feature map with a size of 32x32x128;

[0030] Step d, input the feature map obtained in step c to the fully connected layer Linear1, input a feature vector with a size of 16x16x256, and output an output feature with a size of 1024;

[0031] Step e, input the output feature with a size of 1024 obtained in step d to the fully connected layer Linear2, input a feature vector with a size of 1024, and output a feature vector with a size of 8.

[0032] Step b is implemented according to the following steps:

[0033] Step b1, perform a 3x3 convolution on the feature map with a size of 64x64x64, with an output channel of 96 and padding of 1, to obtain a feature map X a with a size of 64x64x96;

[0034] Step b2, perform a 3x3 up-sampling on the feature map with a size of 64x64x64, with an output channel of 32 and padding of 1, to obtain a feature map X b with a size of 64x64x32;

[0035] Step b3, concatenate the feature maps X a and X b , and activate them with LeakyReLU with a negative slope of 0.2 to obtain a feature map with a size of 64x64x128;

[0036] Step b4, perform a 3x3 convolution on the feature map obtained in step b3, with an output channel of 128 and padding of 1, and activate it with LeakyReLU with a negative slope of 0.2 to obtain a feature map with a size of 64x64x128;

[0037] Step b5, perform a maximum pooling with a kernel of 2 on the feature map obtained in step b4 to finally obtain a feature map with a size of 32x32x128.

[0038] Step c is implemented according to the following steps:

[0039] Step c1, 3x3 convolution is performed on the feature map with a size of 32x32x128, the output channel is 192, the padding is 1, and a feature map with a size of 32x32x192 is obtained X a′ ;

[0040] Step c2, 3x3 upsampling is performed on the feature map with a size of 32x32x128, the output channel is 64, the padding is 1, and a feature map with a size of 32x32x64 is obtained X b′ ;

[0041] Step c3, connect the feature maps X a′ and X b′ , activate with LeakyReLU with negative slope 0.2, and obtain a feature map with a size of 32x32x256;

[0042] Step c4, 3x3 convolution is performed on the feature map obtained in step c3, the output channel is 256, the padding is 1, and LeakyReLU with negative slope 0.2 is activated to obtain a feature map with a size of 32x32x256;

[0043] Step c5, maximum pooling with a kernel of 2 is performed on the feature map obtained in step c4, and finally a feature map with a size of 16x16x256 is obtained.

[0044] The printing image registration method based on the convolution cross attention mechanism can effectively register two printing images in spatial position, and provides guarantee for defect detection of the printing image in the later period, and improves the defect detection efficiency. The application provides a convolution cross attention mechanism, and the key-value query mechanism of the mechanism can effectively fuse the feature information of two images, correspond the hot spot area features of the two images, and fuse into a feature map, which conforms to the general process of image registration. The printing image registration method based on the convolution cross attention mechanism is provided, first, the cross attention mechanism is designed in the parallel network to process the reference printing image and the printing image to be registered, fuse the features of the two images, and then convolve and upsample the feature maps of the two images in a series, and the designed upsample mode can reduce the loss of features. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a flowchart of the printing image registration method based on the convolution cross attention mechanism of the application;

[0046] Figure 2 is a general architecture schematic diagram of the printing image registration network based on the convolution cross attention mechanism of the application;

[0047] Figure 3 are the to-be-registered printed image, the reference printed image and the registration result in the embodiments of the present application. DETAILED DESCRIPTION

[0048] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0049] The printed image registration method based on the convolution cross-attention mechanism of the present application is shown in the flowchart as Figure 1 , and is implemented according to the following steps:

[0050] Step 1, constructing a deep learning registration network, including a convolution cross-attention mechanism and a deep homography estimation registration network based on upsampling;

[0051] The convolution cross-attention mechanism in step 1 is implemented according to the following steps:

[0052] Step 1.1, input two tensors X1, X2∈R H×W×C , H represents the height of the input feature map, W represents the width of the input feature map, C represents the channel number of the input feature map, and the size of X1 and X2 is 64x64x32;

[0053] Step 1.2, in order to ensure that the image processing contains translation invariance, the existing relative position encoding is extended to two dimensions, the relative position embedding of the cross-attention is expanded to two dimensions, that is, the width information and the height information are embedded, two-dimensional relative cross-attention is realized, and the attention degree of pixel i=(i x , i y ) to pixel j=(j x , j y ) is calculated as formula (1):

[0054]

[0055] wherein, l i，j represents the attention degree of pixel i=(i x , i y ) to pixel j=(j x , j y ), represents the transpose of the pixel i query vector, represents the depth of the key k, k j is the key vector of pixel j, and represent the relative width j x -i x and the relative height j y -i y ;

[0056] The output of the two-dimensional single-head cross-attention is formula (2):

[0054]

[0055] wherein, l i，j represents the attention degree of pixel i=(i x , i y ) to pixel j=(j x , j y ), represents the transpose of the pixel i query vector, represents the depth of the key k, k j is the key vector of pixel j, and represent the relative width j x -i x and the relative height j y -i y ;

[0056] The output of the two-dimensional single-head cross-attention is formula (2):

[0057]

[0058] where O h represents the output of two-dimensional single-head cross-attention, softmax(·) represents normalization, W Q represents the weight of the query, W k represents the weight of the key, and W v represents the weight of the value. represents the logical matrix of the relative position of width and height, X1 represents the tensor form of the feature Figure 1 , and X2 represents the tensor form of the feature Figure 2 . represents the depth of the key k.

[0059] Step 1.4, multi-head attention is spliced by single-head attention, as shown in formula (3):

[0060] MHA(X)=Concat[O1,...,O Nh ]W O (3)

[0061] where MHA(X) represents a multi-head attention tensor with a shape of (H, W, d v ), Concat[·] represents splicing, O1,..., O Nh represent single-head attention, represents a weight vector.

[0062] Step 1.5, the convolution and multi-head cross-attention feature map are mapped and connected to obtain the convolution cross-attention, which can be written as formula (4):

[0063] AAConv(X)=Concat[Conv(X),MHA(X)] (4)

[0064] where AAConv(X) represents convolution cross-attention, Concat[·] represents splicing, Conv(X) represents convolution, and MHA(X) represents a multi-head attention tensor with a shape of (H, W, d v ).

[0065] Step 1.6, the convolution cross-attention is batch normalized to obtain a feature map X'1 that integrates X2 features, and the size of X'1 is 64x64x32. Replace X1 with X'1.

[0066] The registration network based on the depth homography estimation of the upsampling in step 1 is implemented according to the following steps:

[0067] Step a, concatenate the feature map tensor X1 and the feature map tensor X2 obtained in step 1 to obtain a single feature map with a size of 64x64x64;

[0068] Step b, perform a transformation operation on the feature map with a size of 64x64x64;

[0069] Step c, perform a transformation operation on the feature map with a size of 32x32x128;

[0070] Step d, input the feature map obtained in step c to the fully connected layer Linear1, input a feature vector with a size of 16x16x256, and output an output feature with a size of 1024;

[0071] Step e, input the output feature with a size of 1024 obtained in step d to the fully connected layer Linear2, input a feature vector with a size of 1024, and output a feature vector with a size of 8.

[0072] Step b is implemented according to the following steps:

[0073] Step b1, perform a 3x3 convolution on the feature map with a size of 64x64x64, with an output channel of 96 and padding of 1, to obtain a feature map X a with a size of 64x64x96;

[0074] Step b2, perform a 3x3 up-sampling on the feature map with a size of 64x64x64, with an output channel of 32 and padding of 1, to obtain a feature map X b with a size of 64x64x32;

[0075] Step b3, concatenate the feature maps X a and X b , and activate them with LeakyReLU with a negative slope of 0.2 to obtain a feature map with a size of 64x64x128;

[0076] Step b4, perform a 3x3 convolution on the feature map obtained in step b3, with an output channel of 128 and padding of 1, and activate it with LeakyReLU with a negative slope of 0.2 to obtain a feature map with a size of 64x64x128;

[0077] Step b5, perform a maximum pooling with a kernel of 2 on the feature map obtained in step b4 to finally obtain a feature map with a size of 32x32x128.

[0078] Step c is implemented according to the following steps:

[0079] Step c1, 3x3 convolution is performed on the feature map with a size of 32x32x128, the output channel is 192, the padding is 1, and a feature map with a size of 32x32x192 is obtained a′ ;

[0080] Step c2, 3x3 upsampling is performed on the feature map with a size of 32x32x128, the output channel is 64, the padding is 1, and a feature map with a size of 32x32x64 is obtained b′ ;

[0081] Step c3, connect the feature maps X a′ and X b′ , activate with LeakyReLU with negative slope of 0.2, and obtain a feature map with a size of 32x32x256;

[0082] Step c4, 3x3 convolution is performed on the feature map obtained in step c3, the output channel is 256, the padding is 1, and LeakyReLU with negative slope of 0.2 is used for activation, and a feature map with a size of 32x32x256 is obtained;

[0083] Step c5, maximum pooling with a kernel of 2 is performed on the feature map obtained in step c4, and finally a feature map with a size of 16x16x256 is obtained.

[0084] Step 2, input the patch (p B ) of the reference printed image and the patch (p A ) of the printed image to be registered into the deep learning registration network, predict the four corner point offset H′ A of p B relative to the four corner points of the reference printed image p 4pt , and obtain the transformation matrix H′ by DLT direct linear transformation;

[0085] Step 3, perform spatial transformation on the transformation matrix H′ and the printed image to be registered A to obtain the registered printed image;

[0086] Step 4, optimize the network parameters by calculating the loss function between the registered printed image and the reference printed image, and output a printed image with higher accuracy.

[0087] Example 1

[0088] As Figure 1 , the printed image registration method based on convolution cross attention mechanism specifically includes:

[0089] Step 1, construct a deep learning registration network, which includes a convolution cross attention mechanism and a deep homography estimation registration network based on upsampling;

[0090] Step 1.1, tensors X1, X2 ∈ R H×W×C into the parallel network, H represents the height of the input feature map, W represents the width of the input feature map, C represents the number of channels of the input feature map, the size of X1, X2 is 128x128x1.

[0091] In combination Figure 2 , Figure 3 , step 1.2, in the parallel network, X1, X2 are respectively convolved twice with a kernel size of 3x3, the output channel is 32, the step is 1, the padding is 1, and the activation function is LeakyReLU with a negative slope of 0.2, thereby obtaining a feature map with a size of 128x128x32, and then performing maximum pooling on the feature map with a kernel size of 2, finally obtaining a feature map with a size of 64x64x32.

[0092] Step 1.3, input the feature map with a size of 64x64x32 into the convolution cross-attention module, wherein the depth of the key dk=10, the depth of the value dv=1, the number of heads of the multi-head attention Nh=1; the convolution module and the attention module are included, the output channel of the convolution module dv, the kernel size is 3, the step is 1, and the padding is 1; the input and output channels of the attention module are dv, the kernel size is 1, and the step is 1; the convolution module and the attention module are spliced to obtain a feature map with a size of 64x64x32 which integrates the features of X2, the size of X'1 is 64x64x32, and X1 is replaced by X'1;

[0093] Step 1.4, concatenate the feature map tensor X1 obtained in step 1.3 and the feature map tensor X2 input in step 1.1 to obtain a single feature map with a size of 64x64x64.

[0094] Step 1.5, the feature map with a size of 64x64x64 is operated as follows:

[0095] (1) 3x3 convolution is performed on the feature map with a size of 64x64x64, the output channel is 96, the padding is 1, and the feature map X a with a size of 64x64x96 is obtained.

[0096] (2) 3x3 upsampling is performed on the feature map with a size of 64x64x64, the output channel is 32, the padding is 1, and the feature map X b with a size of 64x64x32 is obtained.

[0097] (3) connect the feature maps X a and X bis activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 64x64x128;

[0098] (4) The feature map obtained in (3) is subjected to 3x3 convolution with an output channel of 128 and padding of 1, and is activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 64x64x128;

[0099] (5) The feature map obtained in (4) is subjected to maximum pooling with a kernel of 2, to finally obtain a feature map with a size of 32x32x128.

[0100] Step 1.6, the feature map with a size of 32x32x128 is subjected to the following operations:

[0101] (1) The feature map with a size of 32x32x128 is subjected to 3x3 convolution, with an output channel of 192 and padding of 1, to obtain a feature map X a′ with a size of 32x32x192;

[0102] (2) The feature map with a size of 32x32x128 is subjected to 3x3 up-sampling, with an output channel of 64 and padding of 1, to obtain a feature map X b′ with a size of 32x32x64;

[0103] (3) The feature maps X a′ and X b′ are connected, and are activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 32x32x256;

[0104] (4) The feature map obtained in (3) is subjected to 3x3 convolution with an output channel of 256 and padding of 1, and is activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 32x32x256;

[0105] (5) The feature map obtained in (4) is subjected to maximum pooling with a kernel of 2, to finally obtain a feature map with a size of 16x16x256.

[0106] Step 1.7, the feature map obtained in step 1.6 is input into a fully connected layer Linear1, with an input feature vector with a size of 16x16x256, and an output feature with a size of 1024.

[0107] Step 1.8, the output feature with a size of 1024 obtained in step 1.7 is input into a fully connected layer Linear2, with an input feature vector with a size of 1024, and an output feature vector with a size of 8.

[0108] Step 2, step 1.8, the feature vector with a size of 8 is a deep learning registration network, and p A The four corner point offset H' of the four corner points of the printed image p B 4pt The transformation matrix H' is obtained by direct linear transformation through DLT;

[0109] Step 3, the transformation matrix H' is subjected to spatial transformation with the printed image A to be registered, so as to obtain the registered printed image p' B ;

[0110] Step 4, the network parameters are optimized by calculating the loss function between the registered printed image and the reference printed image, and a printed image with higher accuracy is output as shown in Figure 3 The red color represents the real perspective transformation; the yellow color represents the perspective transformation estimated by the model; the more coincident the two are, the higher the accuracy of the registration is.

[0111] The application provides a printed image registration method based on a convolution cross attention mechanism, which can better complete the registration task and obtain the registered image by improving the unsupervised deep homography estimation model for image registration of printed matter, and has important significance for defect detection of printed matter and improves the defect detection efficiency.

[0112] Embodiment 2

[0113] The printed image registration method based on the convolution cross attention mechanism of the application is shown in the flow chart as Figure 1 , and is implemented according to the following steps:

[0114] Step 1, a deep learning registration network is constructed, including a convolution cross attention mechanism and a deep homography estimation registration network based on upsampling;

[0115] The convolution cross attention mechanism in step 1 is implemented according to the following steps:

[0116] Step 1.1, input two given shape tensors X1 and X2, X1, X2∈R H×W×C , H represents the height of the input feature map, W represents the width of the input feature map, C represents the channel number of the input feature map, and the size of X1 and X2 is 64*64*32;

[0117] Step 1.2, in order to ensure that the image processing contains translation invariance, the existing relative position coding is extended to two dimensions, the relative position embedding of the cross attention is widened and the height information is embedded, two-dimensional relative cross attention is realized, and the pixel i=(i x , i y ) is used to represent the pixel j=(j x , j​y The attention level is calculated using formula (1):

[0118]

[0119] Among them, l i，j Represents pixel i = (i x i y For pixel j = (j x j y ) attention, This represents the transpose of the query vector for pixel i. Denotes the depth of bond k, k j It is the key vector of pixel j. and Represents relative width j x -i x and relative height j y -i y ;

[0120] Step 1.3, the output of the two-dimensional single-head cross attention is formula (2):

[0121]

[0122] Among them, O h This represents the output of a two-dimensional single-head cross-attention function, where softmax(·) represents normalization, and W... Q W represents the query weight. k W represents the weight of the key. v The weight of the value. A logical matrix representing the relative positions of width and height, where X1 represents the feature. Figure 1 The tensor form, X2 represents the feature. Figure 2 tensor form, Indicates the depth of key k.

[0123] Step 1.4: Multi-head attention is composed of single-head attention, as shown in formula (3):

[0124] MHA(X) = Concat[O1, ..., O Nh W O (3)

[0125] Where MHA(X) represents a shape of (H, W, d) v The multi-head attention tensor, Concat[·] denotes concatenation, O1, ..., O Nh This indicates single-head attention. Represents the weight vector;

[0126] Step 1.5: Map and connect the convolutional and multi-head cross-attention feature maps to obtain convolutional cross-attention, which can be written as formula (4):

[0127] AAConv(X)=Concat[Conv(X),MHA(X)] (4)

[0128] Where AAConv(X) represents convolutional cross attention, Concat[·] represents concatenation, Conv(X) represents convolution, and MHA(X) represents a shape of (H, W, d). v Multi-head attention tensor;

[0129] Step 1.6: Perform batch normalization on the convolutional cross attention to obtain feature map X′1 that integrates the features of X2. The size of X′1 is 64×64×32. Replace X1 with X′1.

[0130] Step 1, based on the upsampling depth homography estimation registration network, is implemented according to the following steps:

[0131] Step a: Concatenate the feature map tensor X1 and feature map tensor X2 obtained in step 1 to obtain a single feature map of size 64×64×64;

[0132] Step b: Perform a transformation operation on the feature map of size 64×64×64;

[0133] Step c: Perform a transformation operation on the feature map of size 32×32×128;

[0134] Step d: Input the feature map obtained in step c into the fully connected layer Linear1. The input feature vector is 16×16×256 and the output feature is 1024.

[0135] Step e: Input the output feature vector of size 1024 obtained in step d into the fully connected layer Linear2. The input feature vector is of size 1024, and the output feature vector is of size 8.

[0136] Step 2: Patch(p) the reference printed image B ) and the patch(p) of the printed image to be registered A The input is fed into a deep learning registration network to predict p. A The four corner points relative to the reference printed image p B The offsets H′ of the four corner points on 4pt The transformation matrix H′ is obtained by performing a direct linear transformation using DLT.

[0137] Step 3: Perform a spatial transformation on the transformation matrix H′ and the printed image A to be registered to obtain the registered printed image;

[0138] Step 4, optimize the network parameters by calculating the loss function between the registered printed image and the reference printed image, and output a printed image with higher precision.

[0139] Embodiment 3

[0140] The printed image registration method based on the convolution cross-attention mechanism of the present application is shown in the flow chart as Figure 1 The specific implementation is as follows:

[0141] Step 1, construct a deep learning registration network, including a convolution cross-attention mechanism and a deep homography estimation registration network based on upsampling;

[0142] The deep homography estimation registration network based on upsampling in step 1 is implemented as follows:

[0143] Step a, concatenate the feature map tensor X1 and the feature map tensor X2 obtained in step 1 to obtain a single feature map with a size of 64x64x64;

[0144] Step b, perform a transformation operation on the feature map with a size of 64x64x64;

[0145] Step c, perform a transformation operation on the feature map with a size of 32x32x128;

[0146] Step d, input the feature map obtained in step c to the full connection layer Linear1, input a feature vector with a size of 16x16x256, and output an output feature with a size of 1024;

[0147] Step e, input the output feature with a size of 1024 obtained in step d to the full connection layer Linear2, input a feature vector with a size of 1024, and output a feature vector with a size of 8.

[0148] Step b is implemented as follows:

[0149] Step b1, perform a 3x3 convolution on the feature map with a size of 64x64x64, output a channel of 96, and padding of 1 to obtain a feature map X a with a size of 64x64x96;

[0150] Step b2, perform a 3x3 upsampling on the feature map with a size of 64x64x64, output a channel of 32, and padding of 1 to obtain a feature map X b with a size of 64x64x32;

[0151] Step b3, connect the feature maps X a and Xb , and activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 64x64x128;

[0152] Step b4, performing 3x3 convolution on the feature map obtained in step b3, with an output channel of 128 and padding of 1, and activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 64x64x128;

[0153] Step b5, performing maximum pooling with a kernel of 2 on the feature map obtained in step b4, to finally obtain a feature map with a size of 32x32x128.

[0154] Step c is implemented according to the following steps:

[0155] Step c1, performing 3x3 convolution on the feature map with a size of 32x32x128, with an output channel of 192 and padding of 1, to obtain a feature map X a′ with a size of 32x32x192;

[0156] Step c2, performing 3x3 up-sampling on the feature map with a size of 32x32x128, with an output channel of 64 and padding of 1, to obtain a feature map X b′ with a size of 32x32x64;

[0157] Step c3, connecting the feature maps X a′ and X b′ , and activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 32x32x256;

[0158] Step c4, performing 3x3 convolution on the feature map obtained in step c3, with an output channel of 256 and padding of 1, and activated by LeakyReLU with a negative slope of 0.2, to obtain a feature map with a size of 32x32x256;

[0159] Step c5, performing maximum pooling with a kernel of 2 on the feature map obtained in step c4, to finally obtain a feature map with a size of 16x16x256.

[0160] Step 2, inputting the patch (p B ) of the reference printed image and the patch (p A ) of the printed image to be registered into the deep learning registration network, predicting the offset H′ A of the four corner points of p B relative to the four corner points of the reference printed image p 4pt , and obtaining the transformation matrix H′ by DLT direct linear transformation;

[0161] Step 3, spatially transforming the transformation matrix H' with the to-be-registered printed image A to obtain a registered printed image;

[0162] Step 4, optimizing the network parameters by calculating the loss function between the registered printed image and the reference printed image, and outputting a printed image with higher precision.

Claims

1. A method for image registration of printed materials based on convolutional cross-attention mechanism, characterized in that, The specific steps are as follows: Step 1: Construct a deep learning registration network; The convolutional cross-attention mechanism in step 1 is implemented according to the following steps: Step 1.1: Input two tensors of given shapes. H represents the height of the input feature map, W represents the width of the input feature map, and C represents the number of channels in the input feature map. and The size of each is 64×64×32; Step 1.2: Extend the existing relative position encoding to two dimensions by embedding width and height information into the relative positions of the cross-attention, thus achieving two-dimensional relative cross-attention, pixel-wise. For pixels The attention level is calculated using formula (1): （1） in, Represents pixels For pixels attention, This represents the transpose of the query vector for pixel i. Indicates the depth of key k. It is the key vector of pixel j. and Indicates relative width and relative height ; Step 1.3, the output of the two-dimensional single-head cross attention is formula (2): （2） in, This represents the output of a two-dimensional single-head cross-attention function. softmax (·) indicates normalization. Indicates the query weight. Indicates the weight of the key. The weight of the value. , A logical matrix representing the relative positions of width and height. The tensor form representing feature map 1, The tensor form representing feature map 2, Indicates the depth of bond k; Step 1.4: Multi-head attention is composed of single-head attention, as shown in formula (3): （3） in, Indicates a shape of (H, W, The multi-head attention tensor of ) Indicates splicing, This indicates single-head attention. Represents the weight vector; Step 1.5: Map and connect the convolutional and multi-head cross-attention feature maps to obtain convolutional cross-attention, which can be written as formula (4): （4） in, Indicates convolutional cross attention. Indicates splicing, Represents convolution. Indicates a shape of (H, W, The multi-head attention tensor; Step 1.6: Batch normalize the convolutional cross attention to obtain the fused... Feature map of features , The size is 64×64×32. Replace with ; Step 2: Input the reference printed image and the printed image to be registered into the deep learning registration network, and obtain the transformation matrix through direct linear transformation (DLT). ; Step 3: Transform the matrix A spatial transformation is performed between the image A to be registered and the image A of the printed matter to be registered to obtain the registered printed image; Step 4: Optimize the network parameters by calculating the loss function between the registered printed image and the reference printed image to output a printed image with higher accuracy.

2. The printed image registration method based on convolutional cross-attention mechanism according to claim 1, characterized in that, The deep learning registration network in step 1 includes a convolutional cross-attention mechanism and a deep homography estimation registration network based on upsampling.

3. The printed image registration method based on convolutional cross-attention mechanism according to claim 2, characterized in that, The upsampling-based depth homography estimation registration network in step 1 is implemented according to the following steps: Step a: Convert the feature map tensor obtained in step 1 into a multi-dimensional array. and feature map tensor By concatenating them, a single feature map of size 64×64×64 is obtained; Step b: Perform a transformation operation on the feature map of size 64×64×64; Step c: Perform a transformation operation on the feature map of size 32×32×128; Step d: Input the feature map obtained in step c into the fully connected layer Linear1. The input feature vector is 16×16×256 and the output feature is 1024. Step e: Input the output feature vector of size 1024 obtained in step d into the fully connected layer Linear2. The input feature vector is of size 1024, and the output feature vector is of size 8.

4. The printed image registration method based on convolutional cross-attention mechanism according to claim 3, characterized in that, Step b is implemented in the following steps: Step b1: Perform a 3×3 convolution on the 64×64×64 feature map, with 96 output channels and padding of 1, to obtain a feature map of size 64×64×96. ; Step b2: Upsample the 64×64×64 feature map by 3×3, output 32 channels, and padding to 1 to obtain a feature map of size 64×64×32. ; Step b3: Connect feature maps and Activation with LeakyReLU with a negative slope of 0.2 yields a feature map of size 64×64×128; Step b4: Perform a 3×3 convolution on the feature map obtained in step b3, with 128 output channels and padding of 1. Activate with LeakyReLU with a negative slope of 0.2 to obtain a feature map of size 64×64×128. Step b5: Perform max pooling with kernel 2 on the feature map obtained in step b4, and finally obtain a feature map of size 32×32×128.

5. The printed image registration method based on convolutional cross-attention mechanism according to claim 4, characterized in that, Step c is implemented in the following steps: Step c1: Perform a 3×3 convolution on the feature map of size 32×32×128, with 192 output channels and padding of 1, to obtain a feature map of size 32×32×192. ; Step c2: Upsample the 32×32×128 feature map by 3×3, output 64 channels, and set padding to 1 to obtain a feature map of size 32×32×64. ; Step c3: Connect feature maps and Activation with LeakyReLU with a negative slope of 0.2 yields a feature map of size 32×32×256; Step c4: Perform a 3×3 convolution on the feature map obtained in step c3, with 256 output channels and padding of 1. Activate with LeakyReLU with a negative slope of 0.2 to obtain a feature map of size 32×32×256. Step c5: Perform max pooling with kernel 2 on the feature map obtained in step c4, and finally obtain a feature map of size 16×16×256.

Images