End-to-end double-flow seal automatic verification method based on deep learning

Abstract

The invention relates to an end-to-end double-flow seal automatic verification method based on deep learning. The method comprises the steps that an obtained to-be-verified seal image is loaded into a pre-constructed seal verification model, and the seal verification model at least comprises a seal image enhancement network and a double-flow image classification network; when the to-be-verified seal image is verified, image enhancement processing is carried out by using the seal image enhancement network so as to generate the to-be-verified seal image after the image enhancement processing; overlapping the to-be-detected seal image with the extracted reference seal image to generate an overlapped seal image, and performing feature extraction and classification processing on the overlapped seal image by using a double-flow image classification network to generate seal verification information after the feature extraction and classification processing; and based on the seal verification information, determining the authenticity of the to-be-verified seal image. The method can efficiently and accurately identify the authenticity of the seal, and is suitable for seal image authenticity verification tasks of various complex backgrounds.

Description

Technical Field

[0001] The present invention relates to a verification method, in particular to an end-to-end dual-stream seal automatic verification method based on deep learning. Background Art

[0002] As an indispensable credit certificate in business operations and administrative management, seals have legal effect and play a vital role in ensuring the legitimacy of transactions and the validity of documents. However, some lawless elements frequently forge seals in pursuit of improper interests, which not only poses a challenge to seal verification work, but also threatens the social credit system.

[0003] Traditional seal verification methods mostly rely on the judgment of professionals, which is not only time-consuming and labor-intensive, but also susceptible to subjective factors and human errors. In addition, traditional methods lack immediacy and automation capabilities, and therefore cannot efficiently and accurately meet the needs of seal authenticity identification. The emergence of computer vision technology provides a revolutionary solution for automated seal verification.

[0004] When using computer vision technology to verify seals, deep learning and image processing algorithms can be used to enhance seal images and identify authenticity. Specifically, computer vision technology can remove noise interference from seal images and accurately correct images, so that the authenticity of seals can be verified efficiently and accurately to ensure the authenticity and legality of documents. When using computer vision technology to verify seals, it can not only improve the speed and accuracy of seal authenticity verification, but also reduce labor costs and eliminate interference from human factors.

[0005] In addition, when using computer vision to verify seals, it can also adapt to the needs of various application scenarios, such as the need for seal verification in smart contract management and automated document processing. Therefore, computer vision technology has brought new opportunities and prospects for the research of automatic seal verification, and is expected to promote greater progress in this field.

[0006] The application with publication number CN117765561A discloses a seal image authenticity identification method based on Siamese feature extraction neural network, wherein a Siamese model is designed according to the characteristics of the seal image, so that it is suitable for seal image authenticity identification; specifically, the technical core of the application includes: obtaining the seal image to be tested and the corresponding reference image, the reference image is a true seal image; registering and aligning the acquired images; extracting features of the registered and aligned seal image to be tested and the reference image through a Siamese-based feature extraction neural network, and obtaining deep learning features of the seal image to be tested and the reference image; wherein the Siamese-based feature extraction neural network includes at least two feature extraction subnetworks interconnected in the middle layer through an SE-Block structure, and the feature extraction subnetwork is a DenseNet network structure; similarity measurement is performed based on the extracted deep learning features of the seal image to be tested and the reference image; and the authenticity of the seal image to be tested is identified based on the similarity. The invention can perform real-time intelligent identification of the seal image to be tested, determine whether it is a true seal, and realize fast and effective seal authenticity identification.

[0007] However, this application suffers from low accuracy and insufficient robustness when identifying authentic seal images with slight differences; in particular, for seal image identification tasks with complex background interference, the DenseNet network used has low accuracy and is difficult to meet actual application needs.

[0008] In summary, the existing seal verification methods have the following main shortcomings: the manual verification process is cumbersome and subjective, and is not suitable for large-scale applications; traditional computer vision technology is not accurate enough in recognizing small differences when identifying the authenticity of seals, and cannot meet the needs of high-precision verification; although the existing deep learning methods have made certain progress in the field of seal verification, they still face the problem of insufficient robustness, making it difficult to accurately verify seals in complex scenarios, and are easily affected by background interference and factors such as the size and direction of the stamping pressure. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an end-to-end dual-stream seal automatic verification method based on deep learning, which can efficiently and accurately identify the authenticity of seals, and also has strong anti-interference ability, and is suitable for seal image authenticity verification tasks in various complex backgrounds.

[0010] According to the technical solution provided by the present invention, an end-to-end dual-stream seal automatic verification method based on deep learning, the seal automatic verification method comprises:

[0011] Acquire a seal image to be verified, and extract a corresponding reference seal image from a seal verification database based on the seal name of the seal image to be verified, wherein the reference seal image is at least consistent with the seal name of the seal image to be verified;

[0012] The acquired seal image to be verified is loaded into a pre-built seal verification model, so as to verify the seal image to be verified using the seal verification model, wherein:

[0013] The seal verification model at least includes a seal image enhancement network and a two-stream image classification network;

[0014] When verifying the seal image to be verified, the seal image enhancement network is first used to perform image enhancement processing to generate the seal image to be verified after the image enhancement processing;

[0015] Superimposing the seal image to be inspected with the reference seal image extracted above to generate a superimposed seal image, and performing feature extraction and classification processing on the superimposed seal image using a two-stream image classification network to generate seal verification information after the feature extraction and classification processing;

[0016] Based on the seal verification information, the authenticity of the seal image to be verified is determined.

[0017] When the seal image enhancement network is used to perform image enhancement processing on the seal image to be verified, it includes image denoising processing and / or seal correction and registration processing, wherein:

[0018] When the image enhancement processing of the seal image to be verified includes image denoising processing and seal correction processing, the seal image enhancement network includes an enhancement network input processing module, a denoiser and a registration and alignment module connected in sequence, so as to perform image denoising processing and seal correction and registration processing on the seal image to be verified in sequence by using the seal image enhancement network, wherein:

[0019] The seal image to be verified is loaded into the denoiser using the enhanced network input processing module;

[0020] Performing image denoising processing on the seal image to be verified by using a denoiser, so as to generate a denoised seal image to be verified after the image denoising processing;

[0021] The registration and alignment module is used to perform seal correction and registration processing on the denoised seal image to be verified, so as to generate the seal image to be verified after the seal correction and registration processing.

[0022] The denoiser comprises a multi-level residual dense connection module for denoising the seal image to be verified, an up-sampling module for increasing the image resolution and an image quality enhancement module for improving the image quality, wherein:

[0023] The multi-level residual dense connection module, the upsampling module and the image quality enhancement module are connected in sequence, the multi-level residual dense connection module is connected to the output end of the enhancement network input processing module, and the image quality enhancement module is adaptively connected to the registration alignment module;

[0024] The multi-level residual dense connection module includes a feature enhancement basic unit and a densely connected convolutional layer, wherein the feature enhancement basic unit includes a plurality of feature enhancement basic modules connected in series in sequence;

[0025] The feature enhancement basic unit is connected to the densely connected convolutional layer, the output of the densely connected convolutional layer and the output of the enhanced network input processing module are both adaptively connected to the densely connected splicer, and the output end of the densely connected splicer is adaptively connected to the upsampling module.

[0026] Any feature enhancement basic module includes a dense block unit group and a basic module residual scaling unit adaptively connected to the dense block unit group, wherein the dense block unit group includes a plurality of dense block units connected in series in sequence,

[0027] The dense block unit includes a first branch unit, a second branch unit, and a dense block unit intra-splicing device for performing channel splicing on the first branch unit and the second branch unit;

[0028] The first branch unit includes a dense block and a dense block residual scaling unit adaptively connected to the dense block;

[0029] The input of the dense block in the first branch unit is directly loaded into the splicer in the dense block unit via the second branch unit;

[0030] For two adjacent dense block units, along the serial connection direction of the dense block units, the output end of the splicer in the dense block unit of the previous dense block unit is adaptively connected to the first branch unit and the second branch unit in the next dense block unit.

[0031] When the registration and alignment module performs the seal correction and registration processing on the denoised seal image to be verified, it includes:

[0032] Extracting the center point of the pattern of the stamp image to be verified after denoising;

[0033] Generate a registration control point A and a registration control point B based on the pattern center point of the denoised seal image to be verified and the key contour point of the central pattern of the denoised seal image to be verified;

[0034] Based on the pattern center point, configuration control point A and registration control point B, the correction registration angle of the stamp image to be verified after denoising is generated.

[0035] Wherein, the correction registration angle is:

[0036]

[0037] Among them, θ jz is the correction registration angle, θ B is the polar coordinate angle of the registration control point B, θ A is the polar coordinate angle of the registration control point A;

[0038] Based on the correction registration angle, the denoised seal image to be verified is rotated to a target state, so as to generate the seal image to be verified after being rotated to the target state.

[0039] The dual-stream image classification network includes a dual-stream backbone network and a classification network output module connected in sequence, wherein:

[0040] The dual-stream backbone network includes an EfficientNet B0 module for global feature and local feature extraction, an SVIT module for global dependency extraction, and a backbone network channel splicer for channel splicing.

[0041] For the superimposed seal image, the EfficientNet B0 module is used to extract the superimposed seal global features and superimposed seal local features, and the SVIT module is used to extract the superimposed seal global dependency relationship;

[0042] The global features, local features and global dependencies of the superimposed seals are channel-spliced ​​through the backbone network channel splicer, and classified and identified output is performed through the classification network output module to obtain seal verification information after the classification and identification output.

[0043] The EfficientNet B0 module includes a first convolutional layer for feature extraction, an automatic inverted residual bottleneck convolution unit group, a second convolutional layer for feature extraction, a pooling layer, and a global extraction fully connected layer, which are connected in sequence.

[0044] When extracting the superimposed seal global features and the superimposed seal local features of the superimposed seal image, the superimposed seal image is processed sequentially by a feature extraction first convolution layer, an automatic inverted residual bottleneck convolution unit group, a feature extraction second convolution layer, a pooling layer, and a global extraction fully connected layer, and the global extraction fully connected layer outputs the superimposed seal global features and the superimposed seal local features;

[0045] The automatic inverted residual bottleneck convolution unit group includes a plurality of automatic inverted residual bottleneck convolution units, and the automatic inverted residual bottleneck convolution units are connected in series in sequence.

[0046] The automatic inverted residual bottleneck convolution basic unit includes a first convolution layer in the bottleneck convolution, a depth-separable convolution layer, a compression excitation module, a second convolution layer in the bottleneck convolution, a bottleneck convolution Dropout layer and a bottleneck convolution adder, wherein:

[0047] An input end of the first convolution layer in the bottleneck convolution and a first adding end of the bottleneck convolution adder are connected to each other to serve as an input end of the automatic inversion residual bottleneck convolution unit;

[0048] The output of the first convolutional layer in the bottleneck convolution is batch normalized and then loaded into the depthwise separable convolutional layer;

[0049] The output of the depthwise separable convolutional layer is loaded into the compression excitation module after batch normalization, and the output end of the compression excitation module is connected to the second adding end of the bottleneck convolution adder through the second convolutional layer in the bottleneck convolution and the bottleneck convolution Dropout layer;

[0050] The output end of the bottleneck convolution adder is used as the output end of the automatic inversion residual bottleneck convolution basic unit.

[0051] The SVIT module includes a block embedder, a position embedder, a Transformer encoder, and a module layer normalization unit connected in sequence, wherein:

[0052] When extracting the global dependency of the superimposed seal image, the superimposed seal image is processed by the block embedder, the position embedder, the Transformer encoder, and the module layer normalization unit in sequence, where:

[0053] The superimposed seal image is divided by using a block embedder to generate a number of seal image blocks after the division, and the information of the seal image is converted into a high-dimensional embedding representation to realize the linear projection of the seal image block;

[0054] The position embedder is used to determine the position information of each seal image block, and all the seal image blocks are connected in sequence to form an image block feature sequence;

[0055] The image block feature sequence is processed using the Transformer encoder, and the global dependency of the superimposed seal is generated through the module layer normalization unit.

[0056] The Transformer encoder includes a plurality of Transformer encoding units connected in series, wherein:

[0057] For any two serially connected Transformer coding units, along the serial connection direction of the Transformer coding units, the output end of the previous Transformer coding unit is connected to the input end of the next Transformer coding unit;

[0058] For any Transformer encoding unit, including multi-head attention mechanism unit and multi-layer perceptron module;

[0059] The input end of the multi-head attention mechanism unit is used as the input end of the Transformer encoding unit, and the multi-head attention mechanism unit is adaptively connected to the multi-layer perceptron module;

[0060] The input end and the output end of the multi-head attention mechanism unit are respectively connected to the corresponding input end of the first adder of the encoder, and the output end of the first adder of the encoder is connected to the input end of the multi-layer perceptron module.

[0061] The output end of the first adder of the encoder and the output end of the multilayer perceptron module are connected to the corresponding input end of the second adder of the encoder, and the output end of the second adder of the encoder is connected as the output end of the Transformer encoding unit.

[0062] The advantages of the present invention are as follows: a seal verification model is pre-constructed, and the seal image to be verified is verified by using the seal verification model; when verifying the seal image to be verified, a seal image enhancement network is first used to perform image enhancement processing, so as to generate a seal image to be checked after the image enhancement processing; the seal image to be checked is superimposed with the reference seal image extracted above to generate a superimposed seal image, and a dual-stream image classification network is used to perform feature extraction and classification processing on the superimposed seal image, so as to generate seal verification information after the feature extraction and classification processing; based on the seal verification information, the authenticity of the seal image to be verified is determined, the authenticity of the seal can be efficiently and accurately identified, and it also has strong anti-interference ability, and is suitable for seal image authenticity verification tasks with various complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a flow chart of an embodiment of the automatic seal verification of the present invention.

[0064] Figure 2 This is a structural block diagram of an embodiment of the seal image enhancement network of the present invention.

[0065] Figure 3 This is a structural block diagram of an embodiment of a feature enhancement basic module of the present invention.

[0066] Figure 4 This is a structural block diagram of an embodiment of a dual-stream image classification network of the present invention.

[0067] Figure 5 This is a structural block diagram of an embodiment of the automatic inverted residual bottleneck convolution unit of the present invention.

[0068] Figure 6 The following is a working flow chart of an embodiment of the SVIT module of the present invention.

[0069] Figure 7A structural block diagram of an embodiment of the Transformer encoder of the present invention.

[0070] Figure 8 A schematic diagram of an embodiment of training and generating a seal image enhancement network according to the present invention.

[0071] Fig. 9 A schematic diagram of an embodiment of training and generating a dual-stream image classification network according to the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0073] In order to efficiently and accurately identify the authenticity of a seal and to be applicable to seal image authenticity verification tasks with various complex backgrounds, the present invention provides an end-to-end dual-stream automatic seal verification method based on deep learning. Specifically, the automatic seal verification method includes:

[0074] Acquire a seal image to be verified, and extract a corresponding reference seal image from a seal verification database based on the seal name of the seal image to be verified, wherein the reference seal image is at least consistent with the seal name of the seal image to be verified;

[0075] The acquired seal image to be verified is loaded into a pre-built seal verification model, so as to verify the seal image to be verified using the seal verification model, wherein:

[0076] The seal verification model at least includes a seal image enhancement network and a two-stream image classification network;

[0077] When verifying the seal image to be verified, the seal image enhancement network is first used to perform image enhancement processing to generate the seal image to be verified after the image enhancement processing;

[0078] Superimposing the seal image to be inspected with the reference seal image extracted above to generate a superimposed seal image, and performing feature extraction and classification processing on the superimposed seal image using a two-stream image classification network to generate seal verification information after the feature extraction and classification processing;

[0079] Based on the seal verification information, the authenticity of the seal image to be verified is determined.

[0080] Figure 1A schematic diagram of an embodiment of the present invention for seal image verification is shown in the figure. In the figure, when verifying the seal image, it is necessary to first obtain the seal image to be verified, wherein the seal image to be verified is the seal image whose authenticity is to be identified. The seal image to be verified can be obtained by using existing commonly used methods, such as first stamping the seal image to be verified, and then obtaining it by scanning, or directly obtaining the seal image to be verified by image extraction and other methods. The specific method of obtaining the seal image to be verified can be selected according to needs, and examples will not be given one by one here.

[0081] After obtaining the seal image to be verified, the name of the seal image to be verified can be obtained by using commonly used technical means in the technical field, such as the seal name of the seal image to be verified can be obtained based on the optical character recognition (OCR) method. Specifically, when using the OCR method, the position of the text can be located in the seal image to be verified and the text content can be identified, so that the seal name of the seal image to be verified can be obtained after the text content is identified. The method of determining the seal name of the seal image to be verified using the OCR method can be consistent with the existing one. Of course, other methods can also be used to obtain the seal name of the seal image to be verified. The specific method of obtaining the seal name of the seal image to be verified can be selected according to needs, and no examples will be given here one by one.

[0082] It should be noted that when the authenticity of the seal image to be verified is verified, a reference seal image needs to be used. Therefore, a seal verification library can be constructed according to the actual application scenario, and one or more reference seal images can be pre-stored in the seal verification library, and the corresponding reference seal image can be extracted from the seal verification library by the seal name. It can be seen that the reference seal image is at least consistent with the seal name of the seal image to be verified. It can be understood that the reference seal image can be a standard seal image, which is a clean, clear and complete seal image, and the reference seal image generally corresponds to the real seal, that is, the corresponding reference seal image can be made by the real seal. It can be seen that when the corresponding reference seal image cannot be extracted from the seal verification library according to the seal name of the seal image to be verified, the authenticity verification of the seal image to be verified cannot be performed. At this time, the entire seal authenticity verification process will be exited. During specific implementation, the reference seal image can be produced by using an existing commonly used method, and the produced reference seal image is stored in a seal verification library.

[0083] Depend on Figure 1It can be seen that when verifying the seal image to be verified, the obtained seal image to be verified needs to be loaded into a pre-built seal verification model so as to use the seal verification model to verify the seal image to be verified. In one embodiment of the present invention, the seal verification model at least includes a seal image enhancement network and a dual-stream image classification network; specifically, when the seal verification model includes a seal image enhancement network and a dual-stream classification network, the seal image enhancement network is first used to perform image enhancement processing to generate a seal image to be inspected after the image enhancement processing. The method and process of using the seal image enhancement network to perform image enhancement on the seal image to be verified can be referred to the following description, that is, the method of generating the seal image to be inspected can be referred to the following description.

[0084] The stamp image to be inspected is superimposed with the reference stamp image extracted above to generate a superimposed stamp image. In specific implementation, when generating the superimposed stamp image, the corresponding pixel values ​​of the stamp image to be inspected and the reference stamp image can be superimposed in a corresponding proportion. For example, a feasible superimposition method is: I = λ1I1+λ2I2, wherein λ1 and λ2 are superimposition coefficients, I1 is the pixel value of the stamp image to be inspected, and I2 is the pixel value of the reference stamp image. Generally, the sum of the superimposition coefficient λ1 and the superimposition coefficient λ2 is 1. Generally, the superimposition coefficient λ1 and the superimposition coefficient λ2 can both be 0.5. Of course, other superimposition methods can also be used. The specific superimposition methods used will not be listed one by one, and the superimposition method that can generate the superimposed stamp image shall prevail.

[0085] After obtaining the superimposed seal, the superimposed seal image is subjected to feature extraction and classification processing using a dual-stream image classification network to generate seal verification information after feature extraction and classification processing; at this time, based on the seal verification information, the authenticity of the seal image to be verified is determined, wherein the seal verification information may be a binary classification result of authenticity, wherein when the binary classification result of authenticity is 1, it indicates that the seal image to be verified is true; when the binary classification result of authenticity is 0, it indicates that the seal image to be verified is false. Specifically, determining the authenticity of the seal image to be verified specifically refers to whether the seal image to be verified is consistent with the reference seal image. When the seal image to be verified is consistent with the reference seal image, it indicates that the seal image to be verified is true, otherwise, it indicates that the seal image to be verified is false. The seal image to be verified is consistent with the reference seal image, which specifically refers to that the seal image to be verified and the reference seal image have the same features.

[0086] From the above description, it can be seen that the seal image to be verified can be verified through the seal verification model. When the seal verification model is used for verification, end-to-end verification of the seal image to be verified is realized. It can be seen that the authenticity of the seal can be identified efficiently and accurately, and it is suitable for seal image authenticity verification tasks with various complex backgrounds.

[0087] In one embodiment of the present invention, when the seal image enhancement network is used to perform image enhancement processing on the seal image to be verified, it includes image denoising processing and / or seal correction and registration processing, wherein:

[0088] When the image enhancement processing of the seal image to be verified includes image denoising processing and seal correction processing, the seal image enhancement network includes an enhancement network input processing module, a denoiser and a registration and alignment module connected in sequence, so as to perform image denoising processing and seal correction and registration processing on the seal image to be verified in sequence by using the seal image enhancement network, wherein:

[0089] The seal image to be verified is loaded into the denoiser using the enhanced network input processing module;

[0090] Performing image denoising processing on the seal image to be verified by using a denoiser, so as to generate a denoised seal image to be verified after the image denoising processing;

[0091] The registration and alignment module is used to perform seal correction and registration processing on the denoised seal image to be verified, so as to generate the seal image to be verified after the seal correction and registration processing.

[0092] As can be seen from the above description, the seal image enhancement network can perform image enhancement processing on the seal image to be verified. In specific implementation, the image enhancement processing can be image denoising processing and / or seal correction and registration processing. The following takes the seal image enhancement network performing image denoising processing and seal correction processing on the seal image to be verified as an example to specifically describe the seal image enhancement network.

[0093] When the seal image enhancement network is used to sequentially perform image denoising and seal correction processing on the seal image to be verified, in one embodiment of the present invention, the seal image enhancement network includes an enhancement network input processing module, a denoiser, and a registration and alignment module connected in sequence. Specifically, the enhancement network input processing module can receive the seal image to be verified, and the received seal image to be verified can be loaded into the denoiser after input processing, so as to perform image denoising processing by the denoiser, and finally perform seal correction and registration processing by the registration and alignment module. The enhancement network input processing module, denoiser, and registration and alignment module in the seal image enhancement network will be specifically described below.

[0094] In one embodiment of the present invention, the denoiser comprises a multi-level residual dense connection module for denoising the seal image to be verified, an upsampling module for increasing the image resolution and an image quality enhancement module for improving the image quality, wherein:

[0095] The multi-level residual dense connection module, the upsampling module and the image quality enhancement module are connected in sequence, the multi-level residual dense connection module is connected to the output end of the enhancement network input processing module, and the image quality enhancement module is adaptively connected to the registration alignment module;

[0096] The multi-level residual dense connection module includes a feature enhancement basic unit and a densely connected convolutional layer, wherein the feature enhancement basic unit includes a plurality of feature enhancement basic modules connected in series in sequence;

[0097] The feature enhancement basic unit is connected to the densely connected convolutional layer, the output of the densely connected convolutional layer and the output of the enhanced network input processing module are both adaptively connected to the densely connected splicer, and the output end of the densely connected splicer is adaptively connected to the upsampling module.

[0098] Figure 2 FIG. 1 shows an embodiment of a seal image enhancement network in which an enhancement network input processing module adopts a convolutional layer. Figure 2 In the figure, Conv1 is the convolution layer used by the enhanced network input processing module. Specifically, the enhanced network input processing module can use a convolution layer with a convolution kernel of 3×3, a stride of 1, and a padding size of 1; therefore, the input processing of the seal image to be verified by the enhanced network input processing module is the convolution operation performed on the seal image to be verified by using the convolution layer.

[0099] In a specific implementation, the denoiser may include a multi-level residual dense connection module, an upsampling module and an image quality enhancement module connected in sequence. Figure 2 In the figure, the part between the upsampling module and the enhanced network input processing module is the multi-level residual dense connection module. The multi-level residual dense connection module can be used to denoise the seal image to be verified, the upsampling module can be used to increase the resolution of the seal image after denoising, and the image quality enhancement module can be used to improve the image quality of the seal image processed by the upsampling module.

[0100] In one embodiment of the present invention, the multi-level residual dense connection module may include a feature enhancement processing basic unit and a densely connected convolutional layer. Figure 2 In the figure, the basic module (feature enhancement) is the basic unit of feature enhancement processing, Conv2 is the densely connected convolutional layer, and the densely connected convolutional layer can use a convolutional layer with a convolution kernel size of 3×3. Figure 2In the figure, the multi-level residual dense connection module also includes a dense connection splicer, Cat1 is the dense connection splicer, the output end of the dense connection convolution layer is connected to the dense connection splicer, in addition, the enhanced network input processing module is connected to the input end of the feature enhancement basic unit and is also connected to the dense connection splicer, so that the dense connection splicer is used to perform channel splicing on the output of the enhanced network input processing module and the output of the dense connection convolution layer, and the output end of the dense connection splicer is adaptively connected to the upsampling module to achieve adaptive connection between the multi-level residual dense connection module and the upsampling module.

[0101] In a specific implementation, the densely connected stitcher can perform element-level addition operation on two tensors of the same shape ([number of batch images, channel dimension, feature map height, feature map width]) output by the convolution layer Conv1 and the convolution layer Conv2 to achieve channel stitching; the stitchers of the present invention all adopt the same stitching operation, and the stitching operation methods adopted by other stitchers can refer to the description here.

[0102] The feature enhancement basic unit includes several feature enhancement basic modules connected in series. Figure 2 In the figure, the “8×” of the basic module (feature enhancement) means that the feature enhancement basic unit includes 8 feature enhancement basic modules connected in series, that is, Figure 2 An embodiment in which a feature enhancement basic unit includes 8 feature enhancement basic modules connected in series is shown. Of course, the number of feature enhancement basic modules connected in series in the feature enhancement basic unit can also be selected according to actual needs, which will not be repeated here.

[0103] It can be understood that when the number of feature enhancement basic modules increases, the seal image enhancement network can capture more detailed features and noise patterns when processing images. More feature enhancement basic modules mean more levels and deeper feature extraction capabilities, thereby better learning and identifying noise features in seal images, making the denoising effect more significant.

[0104] Increasing the number of feature enhancement basic modules will also increase the number of parameters and computational complexity of the seal image enhancement network, and increase the time for training and reasoning. A small number of feature enhancement basic modules are suitable for application scenarios with limited resources, and can provide basic denoising effects at a lower computing cost; a large number of feature enhancement basic modules are suitable for scenarios with high requirements for image denoising effects and sufficient computing resources, and can provide better denoising effects at a higher computing cost. In specific implementation, the number of feature enhancement basic modules in the feature enhancement basic unit is preferably 8.

[0105] Generally, multiple serially connected feature enhancement basic modules can adopt the same structural form. Figure 3 An embodiment of a feature-added basic module is shown in FIG. Figure 3 The basic modules of feature enhancement are explained in detail.

[0106] In one embodiment of the present invention, any feature enhancement basic module includes a dense block unit group and a basic module residual scaling unit adaptively connected to the dense block unit group, wherein the dense block unit group includes a plurality of dense block units connected in series in sequence,

[0107] The dense block unit includes a first branch unit, a second branch unit, and a dense block unit intra-splicing device for performing channel splicing on the first branch unit and the second branch unit;

[0108] The first branch unit includes a dense block and a dense block residual scaling unit adaptively connected to the dense block, and the output end of the dense block residual scaling unit is connected to the dense block unit internal splicer;

[0109] The input of the dense block in the first branch unit is directly loaded into the splicer in the dense block unit via the second branch unit;

[0110] For two adjacent dense block units, along the serial connection direction of the dense block units, the output end of the splicer in the dense block unit of the previous dense block unit is adaptively connected to the first branch unit and the second branch unit in the next dense block unit.

[0111] In a specific implementation, the feature enhancement basic module may include a dense block unit group and a basic module residual scaling unit, wherein the dense block unit group includes a plurality of dense block units connected in series. Figure 3 An embodiment of a dense block unit group including three dense block units connected in series is shown, wherein each dense block unit includes a first branch unit, a second branch unit and a dense block unit internal splicer, the first branch unit includes a dense block and a dense block residual scaling unit adaptively connected to the dense block, the dense block is connected to an input end of the dense block unit internal splicer through the dense block residual scaling unit, and the input end of the dense block in the first branch unit is also directly connected to the other input end of the dense block unit internal splicer through the second branch unit, so as to realize channel splicing in the dense block unit internal splicer.

[0112] Figure 3, MJ1, MJ2, and MJ3 are three dense blocks, wherein the dense block unit corresponding to the dense block of MJ1 is the first dense block unit, the dense block unit corresponding to the dense block of MJ2 is the second dense block unit, and the dense block unit corresponding to the dense block of MJ3 is the third dense block unit; specifically, β1, β2, and β3 correspond to three dense block residual scaling units, respectively, wherein the dense residual scaling unit corresponding to β1 is the dense residual scaling unit in the first dense block unit, the dense residual scaling unit corresponding to β2 is the dense residual scaling unit in the second dense block unit, and the dense residual scaling unit corresponding to β3 is the dense residual scaling unit in the third dense block unit. Cat2, Cat3, and Cat4 correspond to dense block unit intra-splicers respectively, among which the dense block unit intra-splicer corresponding to Cat2 is the dense block unit intra-splicer in the first dense block unit, the dense block unit intra-splicer corresponding to Cat3 is the dense block unit intra-splicer in the second dense block unit, and the dense unit intra-splicer corresponding to Cat4 is the dense block unit intra-splicer in the third dense block unit.

[0113] From the above description, we can see that Figure 3 In the feature enhancement basic module shown in , the first dense block unit includes a dense block MJ1, a dense block residual scaling unit β1 and a dense block unit stitcher Cat2, the second dense block unit includes a dense block MJ2, a dense block residual scaling unit β2 and a dense block unit stitcher Cat3, and the third dense block unit includes a dense block MJ3, a dense block residual scaling unit β3 and a dense block unit stitcher Cat4.

[0114] From the above description, it can be seen that in the feature enhancement basic module, the dense block units are connected in series in sequence, and the direction of the dense block units in series is Figure 3 The first dense block unit points in the direction of the third dense block unit. Two adjacent dense block units can be divided into the previous dense block unit and the next dense block unit. If the first dense block unit is connected in series with the second dense block unit, the first dense block unit is the previous dense block unit, and the second dense block unit forms the next dense block unit. Similarly, when the second dense block unit is connected in series with the third dense block unit, the second dense block unit forms the previous dense block unit, and the third dense block unit forms the next dense block unit. Therefore, the previous dense block unit and the corresponding next dense block unit can be determined according to the series connection of the dense block units.

[0115] When connected in series, the output end of the splicer in the dense unit of the previous dense block unit is adaptively connected to the first branch unit and the second branch unit of the next dense block unit. For example, when the first dense block unit is connected in series with the second dense block unit, the output end of the splicer Cat2 in the dense block unit is connected to the input end of the dense block MJ3. From the above description, it can be seen that the output end of the splicer Cat2 in the dense block unit is also connected to the input end of the splicer Cat3 in the dense block unit. The connection conditions when other dense block units are connected in series can be referred to the description here, and examples will not be given one by one here.

[0116] Figure 3 A schematic diagram of an embodiment of a basic module residual scaling unit is also shown. In the figure, the basic module residual scaling unit includes a module residual scaling module and a module unit intra-splicer, wherein β4 is the module residual scaling module, Cat5 is the module unit intra-splicer, and the output end of the module residual scaling module is connected to an input end of the module unit intra-splicer; when the dense block unit group is connected to the basic module residual scaling unit, the output end of the dense block unit group is connected to the input end of the module residual scaling module, and the input end of the dense block unit group is also connected to another input end of the module unit intra-splicer.

[0117] In specific implementation, the dense block residual scaling unit β1, the dense block residual scaling unit β2, the dense block residual scaling unit β3, and the module residual scaling module β4 can adopt the same residual scaling parameters. Generally, the corresponding residual scaling parameters of the dense block residual scaling unit β1, the dense block residual scaling unit β2, the dense block residual scaling unit β3 and the module residual scaling module β4 can be set to 0.2.

[0118] Figure 3 In the embodiment, the output end of the splicer in the dense block unit in the third dense block unit can form the output end of the dense block unit group, and the input end of the dense block in the first dense block unit corresponds to the input end of the dense block unit group. Therefore, the output end of the splicer Cat4 in the dense block unit is connected to the input end of the module residual scaling module β4, and the input end of the dense block MJ1 is also connected to another input end of the splicer Cat5 in the module unit. In specific implementation, the output end of the splicer Cat5 in the module unit forms the output end of the feature enhancement basic module, and the input end of the dense block MJ1 forms the input end of the feature enhancement basic module. When multiple feature enhancement basic modules are connected in series, the output end of the previous feature enhancement basic module is connected to the input end of the next feature enhancement basic module. When multiple feature enhancement basic modules are connected in series, the situation of the previous feature enhancement basic module and the next feature enhancement basic module can refer to the corresponding description of the previous dense block unit and the next dense block unit, which will not be repeated here.

[0119] In specific implementation, the dense blocks in the feature enhancement basic module can generally adopt the same form. For example, the above-mentioned dense blocks MJ1 to MJ3 can adopt the same form. The dense blocks include a dense block first-class sub-unit group and at least one dense block second-class sub-unit. The dense block first-class sub-unit group is adaptively connected to the dense block second-class sub-unit. The dense block first-class sub-unit group includes several dense block first-class sub-units connected in series in sequence. The dense block first-class sub-unit includes a first-class sub-unit convolutional layer and a first-class sub-unit LReLU layer adaptively connected to the first-class sub-unit convolutional layer. The dense block second-class sub-unit includes a second-class sub-unit convolutional layer.

[0120] Specifically, the first type subunit convolution layer and the second type subunit convolution layer can adopt a convolution layer with a convolution kernel of 3×3, a stride of 1, and a padding size of 1, and the first type subunit LReLU layer can adopt default parameters.

[0121] Figure 3 An embodiment of a dense block is also shown in the figure. In the figure, the dense block first-category sub-unit group includes an embodiment of four dense block first-category sub-units connected in series, wherein Conv6 is the first-category sub-unit convolution layer in the first-category sub-unit of the first dense block, LR3 is the first-category sub-unit LReLU layer in the first-category sub-unit of the first dense block, Conv7 is the first-category sub-unit convolution layer in the first-category sub-unit of the second dense block, LR4 is the first-category sub-unit LReLU layer in the first-category sub-unit of the second dense block, Conv8 is the first-category sub-unit convolution layer in the first-category sub-unit of the third dense block, LR5 is the first-category sub-unit LReLU layer in the first-category sub-unit of the third dense block, Conv9 is the first-category sub-unit convolution layer in the first-category sub-unit of the fourth dense block, and LR6 is the first-category sub-unit LReLU layer in the fourth-category sub-unit of the first dense block.

[0122] Specifically, for the input loaded into the dense block, the input passes through a series of convolutional layers, and the number of input channels of each convolutional layer gradually increases, while the number of output channels remains consistent; for example, the number of input channels of the first-class subunit convolutional layer Conv6, the first-class subunit convolutional layer Conv7, the first-class subunit convolutional layer Conv8, and the first-class subunit convolutional layer Conv9 are 64, 128, 192, and 256 respectively, and the number of output channels is 64.

[0123] When connected in series, the first-type subunit LReLU layer LR3 is connected to the first-type subunit convolutional layer Conv7, the first-type subunit LReLU layer LR4 is connected to the first-type subunit convolutional layer Conv8, and the first-type subunit LReLU layer LR5 is connected to the first-type subunit convolutional layer Conv9.

[0124] Figure 3In the figure, Conv10 is the second type subunit convolution layer, and when the first type subunit group of the dense block is adaptively connected to the second type subunit of the dense block, the first type subunit LReLU layer LR6 is connected to the second type subunit convolution layer Conv10. The input end of the first type subunit convolution layer Conv6 forms the input end of the dense block, and the output end of the second type subunit convolution layer Conv10 forms the output end of the dense block.

[0125] Further, Figure 2 An embodiment of an upsampling module is also shown in the figure. The upsampling module may include an upsampling convolution layer and an upsampling LeakyReLU layer. The upsampling convolution layer is connected to the upsampling LeakyReLU layer. In the figure, Conv3 is the upsampling convolution layer. The upsampling convolution layer can adopt a convolution layer with a convolution kernel of 3×3; LRU1 is the upsampling LeakyReLU layer. The upsampling module is connected to the output end of the dense connection stitcher Cat1 through the upsampling convolution layer, and the upsampling module is connected to the image quality enhancement module through the upsampling LeakyReLU layer.

[0126] Further, Figure 2 An embodiment of an image quality enhancement module is also shown, and the image quality enhancement module includes a first convolutional layer for image quality enhancement, a LeakyReLU layer for image quality enhancement, and a second convolutional layer for image quality enhancement. The first convolutional layer for image quality enhancement, the LeakyReLU layer for image quality enhancement, and the second convolutional layer for image quality enhancement are connected in sequence. In the figure, Conv4 is the first convolutional layer for image quality enhancement, LRU2 is the LeakyReLU layer for image quality enhancement, and Conv5 is the second convolutional layer for image quality enhancement. Both the first convolutional layer for image quality enhancement and the second convolutional layer for image quality enhancement can adopt convolutional layers with a convolution kernel of 3×3 size. The image quality enhancement module is connected to the upsampling module through the first convolutional layer for image quality enhancement, and is adaptively connected to the registration and alignment module through the second convolutional layer for image quality enhancement.

[0127] For the above-mentioned denoiser, the multi-level residual dense connection module uses residual connections and dense connections to enable the seal image enhancement network to capture more image details and noise features when denoising. Residual connections can better learn the distribution of noise and denoising methods, while dense connections ensure efficient transmission of features and avoid information loss. The upsampling module formed by combining convolution and non-linear activation functions can improve the resolution of the image after denoising by the multi-level residual dense connection module, while retaining and enhancing the image details. The image quality enhancement module is constructed based on convolution layers and non-linear activation functions. The convolution layer is responsible for extracting high-level features, and the activation function provides non-linear transformation to enhance the contrast and details of the image, making the denoised image clearer and cleaner.

[0128] In one embodiment of the present invention, when the registration and alignment module performs seal correction and registration processing on the denoised seal image to be verified, it includes:

[0129] Extracting the center point of the pattern of the stamp image to be verified after denoising;

[0130] Generate a registration control point A and a registration control point B based on the pattern center point of the denoised seal image to be verified and the key contour point of the central pattern of the denoised seal image to be verified;

[0131] Based on the pattern center point, configuration control point A and registration control point B, the correction registration angle of the stamp image to be verified after denoising is generated.

[0132] Wherein, the correction registration angle is:

[0133]

[0134] Among them, θ jz is the correction registration angle, θ B is the polar coordinate angle of the registration control point B, θ A is the polar coordinate angle of the registration control point A;

[0135] Based on the correction registration angle, the denoised seal image to be verified is rotated to a target state, so as to generate the seal image to be verified after being rotated to the target state.

[0136] Figure 2 The GeomPix registration module in is the registration and alignment module. When performing the seal correction and registration process, it is necessary to first determine the center point and the seal control point. After that, the correction and registration angle is determined based on the determined center point and seal control point, and the correction and registration process is implemented based on the determined correction and registration angle. The specific process of extracting the center point and the registration control point A and the registration control point B is described in detail below.

[0137] Generally, there is a seal pattern in the central area of ​​the seal, which is mostly a five-pointed star. Of course, the pattern in the central area of ​​the seal can also be other shapes. The following takes the five-pointed star pattern as an example to illustrate the process of extracting the central area and aligning control points A and B.

[0138] As can be seen from the above description, the seal image to be verified will be processed by the enhanced network input processing module and the denoiser to obtain the denoised seal image to be verified. For the central area of ​​the denoised seal image to be verified, including the contour points of several seal patterns, the existing commonly used circle fitting method is used for the contour points of all seal patterns to determine the center point of the seal pattern, that is, to determine the center point of the pattern. Specifically, the method and process of determining the center point of the pattern by circle fitting can be consistent with the existing ones.

[0139] After determining the center point of the seal pattern, the Euclidean distance between the center point of the pattern and all the contour points is calculated, so that the key contour points of the seal pattern can be determined based on the calculated Euclidean distance; specifically, since the seal pattern is a five-pointed star, the key contour points are the five vertices of the five-pointed star. When determining the key contour points, the five contour points corresponding to the five Euclidean distances with the center point of the pattern are configured as the key contour points of the center pattern, and the five key contour points of the center pattern form a key contour point group.

[0140] Since there are five central pattern key contour points, the registration control point A and the registration control point B are determined based on the pattern center point and the central pattern key contour point, including:

[0141] Select two central pattern key contour points from the key contour point group, the selected two central pattern key contour points and the pattern center point form a fan-shaped area, and count the pixel values ​​of the fan-shaped area;

[0142] Repeat the above operation of selecting two central pattern key contour points from the key contour point group and counting pixel values, and determine the sector area with the smallest pixel value;

[0143] The two central pattern key contour points corresponding to the fan-shaped area with the minimum pixel value are configured as the registration control point A and the registration control point B.

[0144] From the above description, it can be known that the registration control point A and the registration control point B are the two central pattern key points of the seal pattern, that is, the two vertices of the five-pointed star. After determining the registration control point A and the registration control point B, the coordinates of the registration control point A and the registration control point B in the pixel coordinate system of the seal image to be verified can be determined. Specifically, the coordinates of the registration control point A are (x A ,y A ), the coordinates of the registration control point B are (x B ,y A ), then, the polar coordinate angles corresponding to the registration control point A and the registration control point B can be obtained. Specifically,

[0145]

[0146] Among them, θ B is the polar coordinate angle of the registration control point B, θ A is the polar coordinate angle of the registration control point A.

[0147] After obtaining the polar coordinate angle θ of the registration control point B B And the polar coordinate angle θ of the registration control point A A After that, the correction registration angle can be determined, wherein the correction registration angle is:

[0148]

[0149] In specific implementation, the correction registration angle θ is obtained jz After that, based on the correction registration angle θ jz , the denoised stamp image to be verified is rotated to the target state, so as to generate a stamp image to be inspected after being rotated to the target state. The stamp image to be inspected is Figure 2 The image output by the registration and alignment module.

[0150] In one embodiment of the present invention, the dual-stream image classification network includes a dual-stream backbone network and a classification network output module connected in sequence, wherein:

[0151] The dual-stream backbone network includes an EfficientNet B0 module for global feature and local feature extraction, an SVIT module for global dependency extraction, and a backbone network channel splicer for channel splicing.

[0152] For the superimposed seal image, the EfficientNet B0 module is used to extract the superimposed seal global features and superimposed seal local features, and the SVIT module is used to extract the superimposed seal global dependency relationship;

[0153] The global features, local features and global dependencies of the superimposed seals are channel-spliced ​​through the backbone network channel splicer, and classified and identified output is performed through the classification network output module to obtain seal verification information after the classification and identification output.

[0154] Specifically, the two-stream image classification network includes a two-stream backbone network and a classification network output module. The two-stream backbone network includes an EfficientNet B0 module, an SVIT module, and a backbone network channel splicer. Figure 4 An embodiment of a dual-stream backbone network is shown in FIG. 1 , in which EfficientNet B0 is the EfficientNet B0 module of the dual-stream backbone network, SVIT is the SVIT module, and Cat6 is the backbone network channel splicer.

[0155] When the dual-stream image classification network is used to perform feature extraction and classification processing on the superimposed seal image, the global features and local features of the superimposed seal image are extracted through the EfficientNet B0 module, and the global dependency of the superimposed seal image is extracted through the SVIT module; the superimposed seal global features, local features, and global dependencies are channel-joined through the backbone network channel splicer, and classified and recognized output is performed through the classification network output module to obtain the seal verification information after the classification and recognition output. Figure 5The specific conditions of the EfficientNet B0 module, SVIT (Streamlining Vision Transformer) module and classification network output module of the present invention are explained.

[0156] In one embodiment of the present invention, the EfficientNet B0 module includes a first convolutional layer for feature extraction, an automatic inverted residual bottleneck convolution unit group, a second convolutional layer for feature extraction, a pooling layer, and a global extraction fully connected layer connected in sequence, wherein:

[0157] When extracting the superimposed seal global features and the superimposed seal local features of the superimposed seal image, the superimposed seal image is processed sequentially by a feature extraction first convolution layer, an automatic inverted residual bottleneck convolution unit group, a feature extraction second convolution layer, a pooling layer, and a global extraction fully connected layer, and the global extraction fully connected layer outputs the superimposed seal global features and the superimposed seal local features;

[0158] The automatic inverted residual bottleneck convolution unit group includes a plurality of automatic inverted residual bottleneck convolution units, and the automatic inverted residual bottleneck convolution units are connected in series in sequence.

[0159] Figure 4 In the figure, Conv11 is the first convolutional layer for feature extraction, and the first convolutional layer for feature extraction can use a convolutional layer with a convolutional kernel size of 3×3. Conv12 is the second convolutional layer for feature extraction, and the second convolutional layer for feature extraction can use a convolutional layer with a convolutional kernel size of 1×1. The part between the first convolutional layer for feature extraction and the second convolutional layer for feature extraction is the automatic inversion residual bottleneck convolution unit group. Figure 4 In the figure, QN1 is the global feature extraction fully connected layer, and the layer between the global feature extraction fully connected layer and the second convolutional layer of feature extraction is the pooling layer.

[0160] Specifically, the first convolutional layer of feature extraction can use a convolutional layer with a convolution kernel of 3×3, a stride of 2, and a padding size of 1, so as to quickly reduce the spatial resolution of the superimposed seal image and effectively extract the low-level features of the superimposed seal image. The automatic inversion residual bottleneck convolution unit group gradually extracts higher-level features of the image, retaining important local features and global features. The second convolutional layer of feature extraction uses a convolutional layer with a convolution kernel of 1×1 and a stride of 1 to further integrate the feature map output by the automatic inversion residual bottleneck convolution unit group to extract higher-level features. The pooling layer is used to downsample the feature map, reduce the spatial dimension of the feature map, retain important feature information, and reduce the computational complexity. The global extraction fully connected layer flattens the pooled feature map into a one-dimensional vector.

[0161] In a specific implementation, the automatic inverted residual bottleneck convolution unit group includes a plurality of automatic inverted residual bottleneck convolution units, and the automatic inverted residual bottleneck convolution units are connected in series in sequence. Figure 4 An embodiment of an automatic inverted residual bottleneck convolution unit group is shown in the figure. In the figure, the automatic inverted residual bottleneck unit group includes 7 automatic inverted residual bottleneck convolution units connected in series, wherein MBC1 is the first automatic inverted residual bottleneck convolution unit, MBC2 is the second automatic inverted residual bottleneck convolution unit, MBC3 is the third automatic inverted residual bottleneck convolution unit, MBC4 is the fourth automatic inverted residual bottleneck convolution unit, MBC5 is the fifth automatic inverted residual bottleneck convolution unit, MBC6 is the sixth automatic inverted residual bottleneck convolution unit, and MBC7 is the seventh automatic inverted residual bottleneck convolution unit.

[0162] In a specific implementation, the automatic inverted residual bottleneck convolution unit includes at least one automatic inverted residual concatenated convolution basic unit. Figure 4 In the automatic inverted residual bottleneck convolution unit MBC1, an automatic inverted residual concatenation basic unit is included, an automatic inverted residual bottleneck convolution unit MBC2 includes two automatic inverted residual concatenation basic units, an automatic inverted residual bottleneck convolution unit MBC3 includes two automatic inverted residual concatenation basic units, an automatic inverted residual bottleneck convolution unit MBC4 includes three automatic inverted residual concatenation basic units, an automatic inverted residual bottleneck convolution unit MBC5 includes two automatic inverted residual concatenation basic units, and an automatic inverted residual bottleneck convolution unit MBC6 includes four automatic inverted residual concatenation basic units. The inverted residual splicing convolution basic unit, the automatic inverted residual bottleneck convolution unit MBC7 includes one automatic inverted residual splicing convolution basic unit; wherein, when the automatic inverted residual bottleneck convolution unit includes multiple automatic inverted residual splicing convolution basic units, the multiple automatic inverted residual splicing convolution basic units are connected in series in sequence, such as the automatic inverted residual bottleneck convolution unit MBC6 includes 4 automatic inverted residual splicing convolution basic units, then in the automatic inverted residual bottleneck convolution unit MBC6, 4 automatic inverted residual splicing convolution basic units are connected in series in sequence, and other situations can be referred to the description here, which will not be repeated here.

[0163] In one embodiment of the present invention, the automatic inverted residual bottleneck convolution basic unit includes a first convolution layer in the bottleneck convolution, a depth-separable convolution layer, a compression excitation module, a second convolution layer in the bottleneck convolution, a bottleneck convolution Dropout layer, and a bottleneck convolution adder, wherein:

[0164] An input end of the first convolution layer in the bottleneck convolution and a first adding end of the bottleneck convolution adder are connected to each other to serve as an input end of the automatic inversion residual bottleneck convolution unit;

[0165] The output of the first convolutional layer in the bottleneck convolution is batch normalized and then loaded into the depthwise separable convolutional layer;

[0166] The output of the depthwise separable convolutional layer is loaded into the compression excitation module after batch normalization, and the output end of the compression excitation module is connected to the second adding end of the bottleneck convolution adder through the second convolutional layer in the bottleneck convolution and the bottleneck convolution Dropout layer;

[0167] The output end of the bottleneck convolution adder is used as the output end of the automatic inversion residual bottleneck convolution basic unit.

[0168] Specifically, the automatic inverted residual bottleneck convolution basic unit may include a first convolution layer in the bottleneck convolution, a depth-separable convolution layer, a compression excitation module, a second convolution layer in the bottleneck convolution, a bottleneck convolution Dropout layer, and a bottleneck convolution adder. Figure 5 An embodiment of the automatic inverted residual bottleneck convolution basic unit is shown in the figure, in which Conv14 is the first convolution layer in the bottleneck convolution, SDConv is a depth-wise separable convolution layer, the depth-wise separable convolution layer can use a convolution layer with a convolution kernel size of k×k, SE is a compression excitation module, Conv15 is the second convolution layer in the bottleneck convolution, DP3 is a bottleneck convolution Dropout layer, and AD1 is a bottleneck convolution adder; the first convolution layer in the bottleneck convolution and the second convolution layer in the bottleneck convolution can both use a convolution layer with a convolution kernel size of 1×1.

[0169] In specific implementation, the first convolution layer in the bottleneck convolution can use a convolution layer with a convolution kernel of 1×1 and a stride of 1. The first convolution layer in the bottleneck convolution can increase the number of channels of the feature map, so that the subsequent deep convolution can capture more features. The depth-separable convolution layer is used to extract complex local features. By decomposing the standard convolution into deep convolution and point-by-point convolution, the amount of calculation and parameters is significantly reduced, while retaining important feature information. The compression excitation module is used to adaptively adjust the channel weights of the feature map to enhance important features and suppress irrelevant features. The second convolution layer in the bottleneck convolution compresses the number of channels of the feature map output by the compression excitation module. The bottleneck convolution Dropout layer randomly sets the values ​​of some feature maps to zero, reducing the model's dependence on specific features and enhancing the model's generalization ability. The bottleneck convolution adder adds the input feature map of the module with the feature map processed by the dilated convolution, deep convolution, compression excitation module, etc., helping to maintain the original feature information, solve the gradient disappearance problem, and accelerate the training process.

[0170] right Figure 4 The automatic inverted residual bottleneck convolution unit in each automatic inverted residual bottleneck convolution unit has different k×k convolution layers and magnifications of the depthwise separable convolution layer, such as Figure 4In the figure, the k3×3 in the automatic inverted residual bottleneck convolution unit MBC1 means that the depthwise separable convolution layer in the automatic inverted residual bottleneck convolution unit MBC1 adopts a 3×3 convolution layer; the k3×3 in the automatic inverted residual bottleneck convolution unit MBC2 means that the automatic inverted residual bottleneck convolution unit MBC2 has two automatic inverted residual splicing convolution basic units, and each automatic inverted residual splicing convolution basic unit is composed of 6 layers of depthwise separable convolution layers, wherein the depthwise separable convolution layer of each layer adopts a 3×3 convolution layer. For other situations, please refer to the description here, and no examples will be given one by one.

[0171] In one embodiment of the present invention, the SVIT module includes a block embedder, a position embedder, a Transformer encoder and a module layer normalization unit connected in sequence, wherein:

[0172] When extracting the global dependency of the superimposed seal image, the superimposed seal image is processed by the block embedder, the position embedder, the Transformer encoder, and the module layer normalization unit in sequence, where:

[0173] The superimposed seal image is divided by using a block embedder to generate a number of seal image blocks after the division, and the information of the seal image is converted into a high-dimensional embedding representation to realize the linear projection of the seal image block;

[0174] The position embedder is used to determine the position information of each seal image block, and all the seal image blocks are connected in sequence to form an image block feature sequence;

[0175] The image block feature sequence is processed using the Transformer encoder, and the global dependency of the superimposed seal is generated through the module layer normalization unit.

[0176] Figure 4 An embodiment of the SVIT module is also shown in the figure. In the figure, the SVIT module includes a block embedder, a position embedder, a Transformer encoder and a module-level normalization unit connected in sequence, wherein the block embedder is used to divide the superimposed seal image to generate a number of seal image blocks after the division; the position embedder is used to determine the position information of each seal image block, and all the seal image blocks are connected in sequence to form an image block feature sequence; the Transformer encoder is used to process the image block feature sequence, and the module-level normalization unit generates the superimposed seal global dependency.

[0177] Figure 4An embodiment of a block embedder is shown in FIG. The block embedder includes a block embedding convolution layer and a rearrangement operation layer. The block embedding convolution layer is connected to the rearrangement operation layer. The input end of the block embedding convolution layer serves as the input end of the block embedder, and the output end of the rearrangement operation layer serves as the output end of the block embedder. In the figure, Conv13 is a block embedding convolution layer. The block embedding convolution layer can adopt a convolution layer with a convolution kernel size of 4×4 and a stride of 4. It divides the superimposed seal image into multiple non-overlapping image blocks, that is, multiple seal image blocks are obtained. At the same time, each seal image block is mapped to a high-dimensional feature space through a block embedding convolution layer, and the information of the superimposed seal image is converted into a high-dimensional embedding representation to enhance the expression ability of the feature, that is, to realize the linear projection of the seal image block. The rearrangement operation layer flattens the feature map output by the block embedding convolution layer, that is, flattens the seal image block, and converts the spatial structure into a sequence form to meet the needs of the Transformer encoder for subsequent encoding processing. The rearrangement operation layer can adopt an existing commonly used form.

[0178] The position embedder includes a token mechanism module and a position embedding module, wherein when the position embedder is connected to the block embedder, the output end of the rearrangement operation layer is connected to the input end of the token mechanism module, the token mechanism module is connected to the position embedding module, and the position embedding module is connected to the Transformer encoder. Specifically, the token mechanism module is a learnable parameter for capturing the global information of the feature map, which is spliced ​​with the seal image block in the forward propagation of the SVIT module to guide the SVIT module to perform the authenticity classification task of the seal image to be verified. The position embedding module provides position information for each seal image block to ensure that the SVIT module can use the spatial structure of the image for effective feature learning.

[0179] In one embodiment of the present invention, the Transformer encoder includes a plurality of Transformer encoding units connected in series, wherein:

[0180] For any two serially connected Transformer coding units, along the serial connection direction of the Transformer coding units, the output end of the previous Transformer coding unit is connected to the input end of the next Transformer coding unit;

[0181] For any Transformer encoding unit, including multi-head attention mechanism unit and multi-layer perceptron module;

[0182] The input end of the multi-head attention mechanism unit is used as the input end of the Transformer encoding unit, and the multi-head attention mechanism unit is adaptively connected to the multi-layer perceptron module;

[0183] The input end and the output end of the multi-head attention mechanism unit are respectively connected to the corresponding input end of the first adder of the encoder, and the output end of the first adder of the encoder is connected to the input end of the multi-layer perceptron module.

[0184] The output end of the first adder of the encoder and the output end of the multilayer perceptron module are connected to the corresponding input end of the second adder of the encoder, and the output end of the second adder of the encoder is connected as the output end of the Transformer encoding unit.

[0185] In a specific implementation, the Transformer encoder may include a plurality of Transformer encoding units connected in series. Figure 4 An embodiment of a Transformer encoder is shown in FIG. 1 , in which the Transformer encoder may include 6 Transformer encoding units connected in series, that is, Figure 4 The "6×" in the Transformer encoder indicates that the Transformer encoder includes an embodiment of 6 Transformer encoding units connected in series. In addition, the number of Transformer encoding units can be selected according to actual needs. For example, the number of Transformer encoding units can be more or less than 6. The number of Transformer encoding units shall be based on what can meet the actual application requirements.

[0186] Specifically, increasing the number of Transformer encoding units usually enhances the feature expression ability of the SVIT module. Each Transformer encoding unit performs feature transformation through the self-attention mechanism and feedforward neural network, which can capture more complex contextual information and high-level feature representation. As the number of Transformer encoding units increases, the model can learn more details and global dependencies, thereby improving the understanding and processing capabilities of image data.

[0187] Figure 7 , a structural block diagram of an embodiment of a Transformer encoder is shown in FIG. In the figure, each Transformer encoding unit includes a multi-head attention mechanism unit, a first encoder adder, a multi-layer perceptron module (MultilayerPerceptron, MLP) and a second encoder adder. The multi-head attention mechanism unit can adopt an existing commonly used form. The multi-layer perceptron module includes a first perceptron linear layer, a GELU layer, a second perceptron linear layer and a perceptron Dropout layer.

[0188] Figure 7An embodiment of a multilayer perceptron module is shown in FIG. In the figure, LN1 is the first linear layer of the perceptron, LN2 is the second linear layer of the perceptron, and DP1 is the Dropout layer of the perceptron. The input end of the first linear layer of the perceptron serves as the input end of the multilayer perceptron module, and the Dropout layer of the perceptron serves as the output end of the multilayer perceptron module. Therefore, when the output end of the first adder of the encoder is connected to the input end of the multilayer perceptron module, it means that the output end of the first adder of the encoder is connected to the input end of the first linear layer of the perceptron; when the output end of the multilayer perceptron module is connected to the corresponding input end of the second adder of the encoder, it specifically means that the output end of the Dropout layer of the perceptron is connected to the corresponding input end of the second adder of the encoder.

[0189] Figure 6 This is a schematic diagram of an embodiment of the SVIT module of the present invention. As can be seen from the above description, the superimposed seal image will be processed by the block embedder, the position embedder, the Transformer encoder and the module layer normalization layer in sequence. Figure 4 and Figure 6 In the figure, CG1 is the module layer normalization layer.

[0190] Specifically, the block embedding convolutional layer in the block embedder is used to divide the superimposed seal image, and the input image is divided into multiple non-overlapping image blocks. Each image block is mapped to a high-dimensional feature space through a convolutional layer, and the information of the input image is converted into a high-dimensional embedding representation. Finally, the image blocks output by the block embedding convolutional layer are converted into a sequence format through a rearrangement operation layer, wherein the rearrangement operation layer converts the image blocks output by the block embedding convolutional layer into a sequence format, that is, it mainly implements flattening processing, which is mainly to meet the effective processing and analysis of the Transformer encoder.

[0191] Specifically, when the block embedder performs block division on the superimposed seal image, the superimposed seal image is divided into N 4x4 seal image patches to provide a finer-grained feature representation so that the subtle features of the superimposed seal image can be more easily captured. Figure 6 The block-based image representation in , means that the superimposed seal image is divided into N seal image blocks by using the block embedder convolution layer, wherein the number N of the seal image blocks is determined by the size of the superimposed seal image and the convolution kernel size of the block embedding convolution layer.

[0192] Figure 6The flattening of the seal image blocks in specifically refers to flattening each seal image block into a one-dimensional vector using a rearrangement operation layer. For each 4×4 seal image block, the length of the flattened vector is 4×4×3=48. The vector of each flattened seal image block is converted into an embedding vector of a fixed dimension (128). Specifically, the shape of the tensor output by the block embedding convolution layer is [16, 128, 100, 100] ([number of batch images, channel dimension, number of image blocks (height direction), number of image blocks (width direction)]), and the seal image blocks are flattened and arranged into [16, 10000, 128] to be converted into a three-dimensional tensor format suitable for the Transformer encoder. From the above description, it can be seen that the shape of the tensor output by the block embedding convolution layer is a four-dimensional vector, and the four-dimensional vector is converted into a three-dimensional vector using a rearrangement operation layer so as to meet the requirements of the Transformer encoder.

[0193] Each seal image block is mapped to a high-dimensional space through the linear projection operation of the convolutional layer of the block embedder, and the seal image block can be represented by a token (each token represents a local area of ​​the superimposed seal image). At the same time, in order to retain the spatial structure information of the superimposed seal image, a position encoding vector is performed through the position embedder to represent the position information of each token. The tokens represented by all seal image blocks are connected in sequence to form an image block feature sequence.

[0194] As can be seen from the above description, the token mechanism is a learnable parameter, through which the category can be embedded. Specifically, a classification vector (tensor shape is [16,1,128]) is added to the one-dimensional vector generated by the rearrangement operation layer to represent the overall information of the image. After adding the learnable category embedding, the shape becomes [16,10001,128], that is, a stamp block embedding vector is formed.

[0195] Position embedding is used to add position information to the seal block embedding vector, which is used to assist in understanding the position of the seal image block in the superimposed seal image, thereby capturing the relative relationship and global context between the seal image blocks. It fully matches the shape of the input block embedding vector, ensuring that the position embedding can be correctly added to each seal image block and classification token. After position embedding processing, the shape of the seal block embedding vector is still [16,10001,128]. It can be seen that the image block feature sequence can be generated through the position embedder.

[0196] The image block feature sequence generated by the position embedder is sent to the Transformer encoder. Each Transformer encoder consists of a multi-head self-attention layer and a multi-layer perceptron module. The multi-head self-attention layer is used to capture the global dependencies in the image block feature sequence. The tokens in the image block feature sequence interact through the self-attention mechanism to obtain rich contextual information to determine the importance of the token in the entire superimposed seal image.

[0197] The multi-layer perceptron module is used to further perform nonlinear transformation and processing on features in the local feature space. In the multi-layer perceptron module, high-dimensional mapping (HMLP) is used, and the hyperparameter γ is introduced to map the input feature map x from m dimensions to a higher-dimensional feature space m*γ. Considering the increase in model calculation amount and without loss of generality, γ=4 is selected. The calculation method of the multi-layer perceptron module is as follows:

[0198]

[0199] Among them, W1 and b1 represent the weight matrix and bias vector of the first linear layer of the perceptron, W2 and b2 represent the weight matrix and bias vector of the second linear layer of the perceptron, x is the feature map input to the first linear layer of the perceptron, and GELU is the operation performed using the GELU layer.

[0200] It can be understood that when high-dimensional mapping is introduced, the dimension of the weight matrix of the linear transformation changes from m to m*γ, and the bias vector changes from 1 to γ, which gives the SVIT module greater freedom and stronger representation ability in the feature learning process, so that it can better adapt to different data distributions and feature patterns, and help the SVIT module learn feature representations with more discriminative and generalized capabilities.

[0201] The classification network output module includes at least a classification network fully connected layer. Figure 4 An embodiment of the classification network output module is also shown. In the figure, QN2 is the classification network fully connected layer. The classification network fully connected layer is connected to the output end of the backbone network channel splicer. The seal authenticity classification information can be output through the classification network fully connected layer, wherein the seal authenticity classification information is the seal verification information.

[0202] It can be seen from the above description that the seal verification model at least includes a seal image enhancement network and a dual-stream image classification network. In order to form the seal verification model, it is generally necessary to construct a seal image enhancement basic network and a dual-stream image classification basic network respectively, so that the seal image enhancement basic network can be trained to generate a seal image enhancement network, and the dual-stream image classification basic network can be trained to generate a dual-stream image classification network. In one embodiment of the present invention, after the seal image enhancement basic network and the dual-stream image classification basic network are constructed, the seal image enhancement basic network and the dual-stream image classification basic network are trained respectively. After the seal image enhancement network and the dual-stream image classification network are trained, the seal image enhancement network and the dual-stream image classification network are connected in series to form the seal verification model.

[0203] The following is a detailed description of the process of training the seal image enhancement basic network and the two-stream image classification basic network, and generating the seal image enhancement network and the two-stream image classification network respectively.

[0204] It is understandable that the constructed seal image enhancement basic network should be consistent with the seal image enhancement network. The seal image enhancement basic network can refer to the above description of the seal image enhancement network. In order to meet the requirements of training and generating the seal image enhancement network, a discriminator should also be set in the seal image enhancement basic network, and the discriminator is connected to the denoiser.

[0205] When training the basic network for seal image enhancement, it is necessary to first construct a seal verification training data set, wherein the seal verification training data set can be constructed based on 9 groups of seal bodies, a total of 18 seal bodies, the 9 groups of seal bodies can be 6 groups of circular seals and 3 groups of elliptical seals, specifically, 1 group of seal bodies includes two seals, and the two seals in 1 group of seal bodies can have the same shape and text, such as both seal bodies are circular seals or both are elliptical seals, but the details of the two seals are different, such as the seal patterns and texts of the two seals have deviations, the deviations can be width, text content and / or position, etc., and the seal pattern can be the five-pointed star mentioned above. Of course, the number of seal bodies used to construct the seal verification training data set can be selected according to actual needs, so as to meet the training needs.

[0206] The seal verification training data set includes several network training seal images, which can be obtained by manually stamping the above-mentioned seal body on a paper document and scanning it. The constructed seal verification training data set may include 739 network training seal images, which may include noisy seal images and their corresponding clean seal images, wherein the noisy seal image generally refers to a seal image with natural text interference and computer-synthesized optical character interference. It should be noted that the network training seal images formed by a group of seal bodies have the same seal image name, and the method for obtaining the seal image name can refer to the above description.

[0207] When training the basic network for seal image enhancement, the Adam optimizer is used, combined with the momentum method and adaptive learning rate adjustment, which can effectively handle the problems of high-dimensional parameter space and sparse gradients. Figure 8 A schematic diagram of an embodiment of the basic network training for seal image enhancement is shown in FIG. Figure 8 In the paper, the denoiser and the discriminator work in parallel and are indirectly connected through the adversarial loss. The denoiser generates training denoised images, and the discriminator receives the denoised images and clean stamp images generated by the denoiser and tries to distinguish between the two. Figure 8 In the figure, the seal training sample is the noisy seal image. The seal training sample is loaded into the denoiser, and the corresponding adversarial loss and perceptual loss are calculated. The discriminator discriminates the training denoised image based on the clean seal image and calculates the corresponding discriminant loss.

[0208] In specific implementation, by minimizing the adversarial loss and the perceptual loss, the weight of the denoiser is updated to make the generated training denoised image as close as possible to the clean seal image, so that it is difficult for the discriminator to distinguish between the generated training denoised image and the clean seal image. By optimizing the adversarial loss, the discriminator's ability to distinguish is improved, and the classification accuracy when identifying the clean seal image and the training denoised image is enhanced.

[0209] Specifically, the calculation method of adversarial loss, perceptual loss and discriminant loss can be consistent with the existing ones. During training, the training status of the seal image enhancement basic network is determined by the discriminant loss. If the discriminant loss is stable and tends to a lower value and no longer decreases or increases significantly, it can be considered that the training of the seal image enhancement basic network has reached the target state. At this time, a seal image enhancement network can be generated based on the seal image enhancement basic network.

[0210] In one embodiment of the present invention, the constructed dual-stream image classification basic network can refer to the description of the dual-stream image classification network described above. In order to meet the requirements of training and generating the dual-stream image classification network, Figure 4, an embodiment of a two-stream image classification basic network further comprising a data distribution adapter and a gradient flip layer is shown, wherein the data distribution adapter comprises a plurality of adapter units connected in sequence, Figure 4 It is shown in FIG. 1 that the data distribution adapter may include four adapter units connected in sequence. Specifically, for any adapter unit, it includes an adapter linear layer, a batch normalization unit, a ReLU unit, and an adapter Dropout layer connected in sequence. Figure 4 In the figure, LN3 is the adapter linear layer and DP2 is the adapter Dropout layer.

[0211] Fig. 9 A schematic diagram of an embodiment of training a dual-stream image classification basic network is shown in the figure, wherein the above-mentioned seal verification training data set can be used to perform network training on the dual-stream image classification basic network. During training, due to the significant differences in data distribution of the 9 pairs of seal bodies, the dual-stream image classification network may have insufficient generalization ability when encountering new seal images. In order to solve this problem, a dual-stream network training sample set and a dual-stream network test set can be constructed based on the seal verification training data set, wherein the network training seal images corresponding to a pair of seal bodies can be used as the dual-stream network training test set, and the network training seal images corresponding to the remaining pairs of seal bodies can be used to construct a dual-stream network training first sample set and a dual-stream network training second sample set. The dual-stream network training first sample set and the dual-stream network training second sample set both include at least one group of network training seal images of elliptical seals, and the number of groups of seal bodies included in the dual-stream network training first sample set and the dual-stream network training second sample set are preferably the same. When 9 groups of seal bodies are used to construct the seal verification training data set, the dual-stream network training first sample set and the dual-stream network training second sample set both include all network training seal images of 4 groups of seal bodies.

[0212] In one embodiment of the present invention, a multi-task learning training method is used to simultaneously train a dual-stream image classification basic network and a data distribution adapter. Through this strategy, the dual-stream image classification network can not only learn the characteristic differences of seals in the authenticity classification task, but also can promote the dual-stream image classification network to capture the common features between different seals through adversarial training of the data distribution adapter, thereby improving the generalization ability of the dual-stream image classification network, so that it can maintain good performance on unseen seal images.

[0213] In order to meet the requirements of the above-mentioned multi-task learning training method, the first sample set of dual-stream network training and the second sample set of dual-stream network training need to be labeled. For example, the labels of all network training seal images in the first sample set of dual-stream network training can be set to "0", and the labels of all network training seal images in the second sample set of dual-stream network training can be set to "1". Specifically, after the first sample set of dual-stream network training and the second sample set of dual-stream network training are labeled, the first sample set of dual-stream network training and the second sample set of dual-stream network training are used to train the feature extractor of the dual-stream image classification basic network through adversarial training of the data distribution adapter. The feature extractor in the dual-stream image classification basic network is composed of the EfficientNet B0 module and the SVIT module, which identifies and learns the common features between different seals and enhances the generalization ability of the model.

[0214] When training the dual-stream image classification basic network, in one embodiment of the present invention, the corresponding network training seal images in the dual-stream network training first sample set and the dual-stream network training second sample set are input into the dual-stream image classification basic network for training, and then the trained dual-stream image classification basic network is tested using the dual-stream network training test set, at which point, one epoch of training is completed. In addition, the Adam optimizer is used for the network training of the dual-stream image classification basic network to achieve efficient feature learning and recognition.

[0215] Specifically, in each epoch iterative training, for all network training seal images in the first sample set of the dual-stream network training, two network training seal images with the same seal name are taken as a first sample image pair each time, and the two network training seal images in the first sample image are respectively used as classification network training seal images and classification network reference seal images, and image preprocessing and image superposition processing are performed on the classification network training seal images and the classification network reference seal images to form the classification network first set training samples, and the formed classification network first set training samples are input into the dual-stream image classification basic network, and thereafter, the corresponding first sample image pair authenticity classification output information is output through the classification network output module, and the first sample image pair prediction information is output through the data distribution adapter.

[0216] Specifically, all the network training seal images in the first sample set of dual-stream network training can form multiple first sample image pairs. It can be understood that the two network training seal images in the multiple first sample image pairs should not be exactly the same. When the constructed first sample image pair is exactly the same as the formed first sample image pair, it indicates that all the network training seal images in the first sample set of the current dual-stream network training have been used. At this time, the training of the dual-stream image classification basic network using the first sample set of the dual-stream network training should be stopped.

[0217] In each epoch of iterative training, for all network training seal images in the second sample set of dual-stream network training, two network training seal images are taken each time as a second sample image pair. Thereafter, the second sample image pair can be used to form the second set of classification network training samples, which are input into the dual-stream image classification basic network, and the second sample image pair prediction information is output through the data distribution adapter. Specifically, the second sample image pair and the second set of classification network training samples can refer to the corresponding description of the first sample image pair and the first set of classification network training samples; it can be understood that based on multiple second sample image pairs, multiple corresponding second sample image pair prediction information can be obtained. That is, different from the first sample set of dual-stream network training, when the dual-stream image classification basic network is trained in the second sample set of dual-stream network training, only the corresponding second sample image pair prediction information through the data distribution adapter can be output.

[0218] Through the true and false classification output information of the first sample image pair, the prediction information of the first sample image pair after the data distribution adapter, and the prediction information of the second sample image pair after the data distribution adapter, the corresponding loss function value after the current epoch iterative training can be calculated. In one embodiment of the present invention, the loss function can adopt VerifiLoss, wherein when the loss function adopts the loss function VerifiLoss, then:

[0219]

[0220] Among them, A_labels is the first label value of the first set of training samples of the classification network, A_output is the prediction information of the first sample image pair of the first set of training samples of the classification network after the data distribution adapter; B_labels is the first label value of the second set of training samples of the classification network, B_output is the prediction information of the second sample image pair of the second set of training samples of the classification network after the data distribution adapter; s_output is the true and false classification output information of the first sample image pair of the first set of training samples of the classification network; s_label is the second label value of the first set of training samples of the classification network.

[0221] It can be seen from the above description that, since the labels of all network training seal images in the first sample set of dual-stream network training can be "0", the first label value A_labels of the first set of classification network training samples can be "0". When the first label value A_labels of the first sample image pair can be 0, the label value B_labels of the second sample image pair should be 1. The second label value s_label of the first set of classification network training samples is related to the two training network seal images that form the first set of classification network training samples. If the two training network seal images come from the same seal body, the second label value s_label of the first set of classification network training samples can be 1, otherwise, the second label value s_label of the first set of classification network training samples is 0.

[0222] In specific implementation, when calculating the loss function VerifiLoss, the second label value s_label of the first set of training samples of the classification network and the first sample image of the first set of training samples of the classification network are used to classify the authenticity output information s_output, so that the trained dual-stream image classification network can recognize and learn the feature differences between authentic and fake seals to realize the authenticity classification of seals.

[0223] When calculating the loss function VerifiLoss, the following takes the calculation of BCELoss (A_output, A_labels) as an example to illustrate the calculation process, specifically:

[0224]

[0225] Where M is the number of training samples in the first set of classification network, y i is the first label value A_labels of the first set of training samples in the current classification network, p i It is the prediction information A_output of the first sample image of the first set of training samples of the current classification network after the data distribution adapter.

[0226] It should be noted that when calculating BCELoss(A_output,A_labels) above, y i Instead of the first label value A_labels, use p i The prediction information A_output of the first sample image pair after the data distribution adapter is only for the convenience of description. The calculation of the above loss function VerifiLoss can refer to the description here and will not be described in detail one by one.

[0227] From the above description, it can be seen that the two-stream image classification basic network may include a gradient flip layer and a data distribution adapter. During each epoch iterative training, in the forward propagation process, the gradient flip layer directly transfers the input features as an identity mapping; in the back propagation process, the gradient flip layer multiplies it by a negative scalar coefficient α to reverse the gradient, thereby applying a negative gradient when optimizing the feature extractor composed of the EfficientNet B0 module and the SVIT module. At this time, the feature extractor can be prompted to ignore the features of specific seals during training and focus on features that are irrelevant to the seal authenticity classification task, thereby weakening the influence of specific features of different seals and enhancing generalization ability.

[0228] In addition, the linear layer in the data distribution adapter linearly transforms the input data through the weight matrix and bias vector to map the input features to the new space. The batch normalization unit standardizes each batch of data so that the output has zero mean and unit variance, thereby accelerating the training process and improving the stability of the two-stream image classification base network. The ReLU unit in the data distribution adapter introduces nonlinearity, allowing the two-stream image classification base network to capture more complex feature relationships. The adapter Dropout layer in the data distribution adapter randomly discards a portion of neurons with a probability of 50% during training, thereby effectively enhancing the generalization ability of the model and reducing the risk of overfitting.

[0229] It is understandable that during the training process, it is necessary to repeat the above process of constructing the first sample set for dual-stream network training, the second sample set for dual-stream network training, and performing an epoch iteration training, that is, it is generally necessary to perform multiple epoch iteration training on the dual-stream image classification basic network. After each epoch iteration training, the loss function value of the loss function VerifiLoss is used to perform back propagation to update the network parameters of the dual-stream image classification basic network.

[0230] During the training process of multiple epoch iterations, by recording and observing the changes in the loss function VerifiLoss during the training process, when the loss function VerifiLoss changes stably and tends to a lower value and no longer decreases or increases significantly, it can be considered that the two-stream image classification basic network has reached the training target state. At this time, the corresponding two-stream image classification network can be generated based on the two-stream image classification basic network that has reached the target state.

[0231] In addition, in the later stage of training, the network parameters of the two-stream image classification basic network of the last few epochs are averaged to smooth parameter changes, reduce fluctuations caused by noise and randomness, and reduce the risk of overfitting, thereby further enhancing the generalization ability of the two-stream image classification basic network. For example, after training for 3 epochs, the loss function VerifiLoss tends to be stable. Therefore, the network parameters of the two-stream image classification basic network of the last 3 epochs are combined to determine the network parameters of the two-stream image classification network. At this time, the overall performance of the two-stream image classification basic network on the two-stream network training test set can be improved.

[0232] From the above description, when the network parameters of the two-stream image classification basic network of the last three epochs are combined to determine the network parameters of the two-stream image classification network, we have:

[0233]

[0234] Where n is the total number of epoch iterations when training the two-stream image classification basic network, current param is the network parameters of the two-stream image classification basic network trained in the i-th epoch iteration, average param It is understandable that the network parameters of the dual-stream image classification basic network are the necessary network parameters for forming the dual-stream image classification network, such as some of the bias parameters and weight parameters mentioned above. The necessary network parameters can be selected according to actual needs and will not be listed here.

[0235] also, Fig. 9 In the preprocessing of the classification network training seal image and the classification network reference seal image, specifically refers to adjusting the corresponding image sizes of the classification network training seal image and the classification network reference seal image to 400×400; when the corresponding images of the classification network training seal image and the classification network reference seal image are adjusted to 400×400, the corresponding sizes of the seal image to be verified and the reference seal image should also be 400×400. Fig. 9 For the image overlay method in , please refer to the above description and will not be repeated here.

[0236] In specific implementation, the two-stream image classification network generated by the above-mentioned two-stream image classification basic network training can be evaluated using the two-stream network training test set. The evaluation indicators of the two-stream network training test set may include precision and recall. The method and purpose of evaluation using precision and recall can be consistent with the existing ones, and will not be repeated here.

Claims

1. An end-to-end dual-stream seal automatic verification method based on deep learning, characterized in that: The automatic seal verification method comprises: Acquire a seal image to be verified, and extract a corresponding reference seal image from a seal verification database based on the seal name of the seal image to be verified, wherein the reference seal image is at least consistent with the seal name of the seal image to be verified; The acquired seal image to be verified is loaded into a pre-built seal verification model, so as to verify the seal image to be verified using the seal verification model, wherein: The seal verification model at least includes a seal image enhancement network and a two-stream image classification network; When verifying the seal image to be verified, the seal image enhancement network is first used to perform image enhancement processing to generate the seal image to be verified after the image enhancement processing; Superimposing the seal image to be inspected with the reference seal image extracted above to generate a superimposed seal image, and performing feature extraction and classification processing on the superimposed seal image using a two-stream image classification network to generate seal verification information after the feature extraction and classification processing; Based on the seal verification information, the authenticity of the seal image to be verified is determined.

2. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 1 is characterized in that: When the seal image enhancement network is used to perform image enhancement processing on the seal image to be verified, it includes image denoising processing and / or seal correction and registration processing, wherein: When the image enhancement processing of the seal image to be verified includes image denoising processing and seal correction processing, the seal image enhancement network includes an enhancement network input processing module, a denoiser and a registration and alignment module connected in sequence, so as to perform image denoising processing and seal correction and registration processing on the seal image to be verified in sequence by using the seal image enhancement network, wherein: The seal image to be verified is loaded into the denoiser using the enhanced network input processing module; Performing image denoising processing on the seal image to be verified by using a denoiser, so as to generate a denoised seal image to be verified after the image denoising processing; The registration and alignment module is used to perform seal correction and registration processing on the denoised seal image to be verified, so as to generate the seal image to be verified after the seal correction and registration processing.

3. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 2 is characterized in that: The denoiser comprises a multi-level residual dense connection module for denoising the seal image to be verified, an up-sampling module for increasing the image resolution and an image quality enhancement module for improving the image quality, wherein: The multi-level residual dense connection module, the upsampling module and the image quality enhancement module are connected in sequence, the multi-level residual dense connection module is connected to the output end of the enhancement network input processing module, and the image quality enhancement module is adaptively connected to the registration alignment module; The multi-level residual dense connection module includes a feature enhancement basic unit and a densely connected convolutional layer, wherein the feature enhancement basic unit includes a plurality of feature enhancement basic modules connected in series in sequence; The feature enhancement basic unit is connected to the densely connected convolutional layer, the output of the densely connected convolutional layer and the output of the enhanced network input processing module are both adaptively connected to the densely connected splicer, and the output end of the densely connected splicer is adaptively connected to the upsampling module.

4. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 3 is characterized in that: Any feature enhancement basic module includes a dense block unit group and a basic module residual scaling unit adaptively connected to the dense block unit group, wherein the dense block unit group includes a plurality of dense block units connected in series in sequence, The dense block unit includes a first branch unit, a second branch unit, and a dense block unit intra-splicing device for performing channel splicing on the first branch unit and the second branch unit; The first branch unit includes a dense block and a dense block residual scaling unit adaptively connected to the dense block; The input of the dense block in the first branch unit is directly loaded into the splicer in the dense block unit via the second branch unit; For two adjacent dense block units, along the serial connection direction of the dense block units, the output end of the splicer in the dense block unit of the previous dense block unit is adaptively connected to the first branch unit and the second branch unit in the next dense block unit.

5. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 2 is characterized in that: When the registration and alignment module performs the seal correction and registration processing on the denoised seal image to be verified, it includes: Extracting the center point of the pattern of the stamp image to be verified after denoising; Generate a registration control point A and a registration control point B based on the pattern center point of the denoised seal image to be verified and the key contour point of the central pattern of the denoised seal image to be verified; Based on the pattern center point, configuration control point A and registration control point B, the correction registration angle of the stamp image to be verified after denoising is generated. Wherein, the correction registration angle is: Among them, θ jz is the correction registration angle, θ B is the polar coordinate angle of the registration control point B, θ A is the polar coordinate angle of the registration control point A; Based on the correction registration angle, the denoised seal image to be verified is rotated to a target state, so as to generate the seal image to be verified after being rotated to the target state.

6. The end-to-end dual-stream seal automatic verification method based on deep learning according to any one of claims 1 to 5, characterized in that: The dual-stream image classification network includes a dual-stream backbone network and a classification network output module connected in sequence, wherein: The dual-stream backbone network includes an EfficientNet B0 module for global feature local feature extraction, an SVIT module for global dependency extraction, and a backbone network channel splicer for channel splicing. For the superimposed seal image, the local features and global features of the superimposed seal are extracted through the EfficientNet B0 module, and the global dependency of the superimposed seal image is extracted through the SVIT module; The backbone network channel splicer is used to perform channel splicing on the superimposed seal local features, superimposed seal global features, and superimposed seal global dependencies, and the classification network output module is used to perform classification and recognition output to obtain seal verification information after the classification and recognition output.

7. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 6 is characterized in that: The EfficientNet B0 module includes a first convolutional layer for feature extraction, an automatic inverted residual bottleneck convolution unit group, a second convolutional layer for feature extraction, a pooling layer, and a global extraction fully connected layer, which are connected in sequence. When extracting the superimposed seal global features and the superimposed seal local features of the superimposed seal image, the superimposed seal image is processed sequentially by a feature extraction first convolution layer, an automatic inverted residual bottleneck convolution unit group, a feature extraction second convolution layer, a pooling layer, and a global extraction fully connected layer, and the global extraction fully connected layer outputs the superimposed seal global features and the superimposed seal local features; The automatic inverted residual bottleneck convolution unit group includes a plurality of automatic inverted residual bottleneck convolution units, and the automatic inverted residual bottleneck convolution units are connected in series in sequence.

8. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 7 is characterized in that: The automatic inverted residual bottleneck convolution basic unit includes a first convolution layer in the bottleneck convolution, a depth-separable convolution layer, a compression excitation module, a second convolution layer in the bottleneck convolution, a bottleneck convolution Dropout layer and a bottleneck convolution adder, wherein: An input end of the first convolution layer in the bottleneck convolution and a first adding end of the bottleneck convolution adder are connected to each other to serve as an input end of the automatic inversion residual bottleneck convolution unit; The output of the first convolutional layer in the bottleneck convolution is batch normalized and then loaded into the depthwise separable convolutional layer; The output of the depthwise separable convolutional layer is loaded into the compression excitation module after batch normalization, and the output end of the compression excitation module is connected to the second adding end of the bottleneck convolution adder through the second convolutional layer in the bottleneck convolution and the bottleneck convolution Dropout layer; The output end of the bottleneck convolution adder is used as the output end of the automatic inversion residual bottleneck convolution basic unit.

9. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 7 is characterized in that: The SVIT module includes a block embedder, a position embedder, a Transformer encoder, and a module layer normalization unit connected in sequence, wherein: When extracting the global dependency of the superimposed seal image, the superimposed seal image is processed by the block embedder, the position embedder, the Transformer encoder, and the module layer normalization unit in sequence, where: The superimposed seal image is divided by using a block embedder to generate a number of seal image blocks after the division, and the information of the seal image is converted into a high-dimensional embedding representation to realize the linear projection of the seal image block; The position embedder is used to determine the position information of each seal image block, and all the seal image blocks are connected in sequence to form an image block feature sequence; The image block feature sequence is processed using the Transformer encoder, and the global dependency of the superimposed seal is generated through the module layer normalization unit.

10. The end-to-end dual-stream seal automatic verification method based on deep learning according to claim 7 is characterized in that: The Transformer encoder includes a plurality of Transformer encoding units connected in series, wherein: For any two serially connected Transformer coding units, along the serial connection direction of the Transformer coding units, the output end of the previous Transformer coding unit is connected to the input end of the next Transformer coding unit; For any Transformer encoding unit, including multi-head attention mechanism unit and multi-layer perceptron module; The input end of the multi-head attention mechanism unit is used as the input end of the Transformer encoding unit, and the multi-head attention mechanism unit is adaptively connected to the multi-layer perceptron module; The input end and the output end of the multi-head attention mechanism unit are respectively connected to the corresponding input end of the first adder of the encoder, and the output end of the first adder of the encoder is connected to the input end of the multi-layer perceptron module. The output end of the first adder of the encoder and the output end of the multilayer perceptron module are connected to the corresponding input end of the second adder of the encoder, and the output end of the second adder of the encoder is connected as the output end of the Transformer encoding unit.

Images