Enhanced entry and exit inspection stamp fine granularity detection method and system based on multi-level feature relationship

Abstract

The invention belongs to the technical field of computers, and discloses an inspection stamp fine-grained detection method and system based on multi-level feature relationship enhancement. Comprising an image acquisition preprocessing module, an image feature extraction network module, a candidate box generation network module, a complementarity multi-level feature fusion network module and a multi-level feature relationship enhancement network module. After an image is input into the network, size normalization processing is firstly carried out, then multi-scale features of an inspection stamp image are obtained through a backbone network, reflection features are further refined through a reflection feature enhancement network fusing channel attention, a probability graph of a reflection region and a prediction graph of a reflection image are output into a generative adversarial network, and the probability graph of the reflection region and the prediction graph of the reflection image are obtained. And image generation is guided by suppressing a reflection dominant region and enhancing a transmission dominant region under the combined action of the generation network and the discrimination network. According to the inspection stamp fine-grained detection method provided by the invention, important breakthroughs are made in key technologies such as feature fusion and discriminative feature selection, and the fine-grained detection and classification accuracy of the inspection stamp fine-grained detection method is superior to that of other existing detection methods in a passport inspection stamp automatic inspection system; and meanwhile, a solid method foundation is laid for further research on the inspection stamp image and more related downstream tasks and applications of the automatic inspection system for the inspection stamp.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a fine-grained detection method for entry-exit inspection stamps based on multi-level feature relationship enhancement. Background Art

[0002] This invention focuses on the field of verification stamp detection technology, aiming to develop a fine-grained verification stamp detection method and system based on multi-level feature relationship enhancement. In practical scenarios such as entry-exit management and customs inspection, accurate detection and classification of verification stamps plays a key role in maintaining border security and ensuring legal entry and exit order. The technical solution provided by this invention enables precise positioning and classification of verification stamps, laying a solid foundation for subsequent in-depth extraction and analysis of verification stamp text information, thereby improving the efficiency and accuracy of the entire inspection process.

[0003] In the verification process for stamps, accurately locating stamps and identifying their categories is a prerequisite for efficient interpretation of the stamp's text. However, current stamp inspections face numerous challenges. Globally, stamp designs in different countries exhibit a high degree of similarity. This requires inspection systems to possess exceptionally high feature resolution capabilities.

[0004] From an entry-exit perspective, entry-exit stamps from the same country are often identical in design, making traditional detection methods prone to misjudgment. Furthermore, stamp types are categorized based on multiple factors, including country, entry-exit type, and time period, resulting in a complex and diverse range of stamp types, further complicating detection and identification.

[0005] Currently, widely used object detection algorithms, such as the Yolo series and Faster R-CNN, have demonstrated excellent performance in conventional natural image object detection tasks, enabling rapid and accurate identification of various common objects. However, when applied to stamp detection scenarios, these algorithms struggle to capture key discriminative features due to the extremely subtle differences between stamp categories. This results in an inability to effectively achieve fine-grained classification while maintaining detection accuracy. Furthermore, the conventional approach of simply splitting fine-grained object detection into two independent sub-processes, detection and classification, introduces two independent sub-networks. This not only prevents network weights from being shared, significantly increasing computational complexity, but also easily leads to information loss and bias during transmission. Furthermore, the method used by existing common object detection algorithms to perform predictions after fusing feature layers within the neck network can easily cause the model to overlook the discriminative features necessary for fine-grained classification in the stamp detection task and fail to fully consider the underlying structural relationships between these discriminative features, severely impacting fine-grained classification performance.

[0006] Through the above analysis, the problems and defects of the existing technology are: the existing methods are difficult to capture these key discriminative features, and fail to fully consider the potential structured relationship between these discriminative features, resulting in the inability to effectively achieve fine-grained classification while ensuring detection accuracy. Summary of the Invention

[0007] In response to the existing technical issues, the core objective of this invention is to provide a fine-grained stamp detection method and system based on multi-level feature relationship enhancement, aiming to fundamentally resolve the challenges of stamp detection and fine-grained classification. Through innovative technical means, the accuracy of stamp detection is significantly improved, enabling precise distinction between different types of stamps. This system also reduces computational complexity and improves overall operational efficiency, thus meeting the high demands of stamp detection in practical application scenarios.

[0008] The present invention is implemented as follows: a fine-grained verification stamp detection method based on multi-level feature relationship enhancement includes the following steps:

[0009] S1: The main body of the present invention adopts a two-stage end-to-end design architecture, pre-processing the collected passport verification stamp image as the input of the model;

[0010] S2: The normalized image data is used as the input of the verification stamp image feature extraction module. The backbone network is used to effectively extract the multi-level image detail features of the input verification stamp image to form a multi-level feature map. The top two levels of the feature map are used as the input of the candidate box network, and the candidate box generation network is used to roughly locate the verification stamp area.

[0011] S3: The multi-level features obtained in S1 are input into the complementary multi-level feature fusion module. The multi-level feature fusion module is designed to integrate feature information at different levels and scales of the image, so that the network can learn more comprehensive, complementary and diverse feature information.

[0012] S4: The fused feature map obtained in S2 is input into the multi-level feature relationship enhancement module. The fine-grained features of the verification stamp are discovered and selected through the mask mechanism, and the multi-level feature relationship is enhanced through cross-attention. At this time, the feature map of the verification stamp contains more discriminative features.

[0013] S5: After obtaining the candidate box predicted by the high-quality candidate box module, the feature map patch obtained in S4 is input into the fine-grained classification regression subnetwork through the RoI Align technology to obtain accurate inspection stamp detection box and fine-grained category information of the inspection stamp.

[0014] Furthermore, step S1 specifically includes:

[0015] The task of stamp detection and fine-grained classification aims to simultaneously locate the stamp area and perform fine-grained classification of the stamp. The current algorithm first locates the stamp area using the stamp detection network, then crops the stamp area and feeds it into the stamp fine-grained classification network. This approach fails to simultaneously complete both localization and fine-grained classification.

[0016] While a one-stage object detection algorithm can achieve better results in common scenarios, it cannot locate the candidate box for the stamp area, making it difficult to extract discriminative features for the stamp area. Therefore, this method improves upon the two-stage object detection network, feeding the image preprocessed with size normalization into the network.

[0017] Further, step S2 specifically includes:

[0018] The pre-processed entry-exit inspection stamp image data H×W obtained by S1, where H and W represent the height and width of the inspection stamp feature map respectively, is input into the image feature extractor; wherein, the image feature extractor is composed of a convolution layer, a ReLU activation layer, an instance normalization layer, etc., and at the same time adopts large kernel convolution and dilated convolution to improve the ability to capture the discriminative shape features of the inspection stamp, which can effectively capture the local detail feature differences of the inspection stamp, improve the representation ability of the discriminative features of the inspection stamp image, and obtain a series of multi-level inspection stamp image feature maps The quality of the candidate frames largely determines the accuracy of subsequent fine-grained classification. For the inspection stamp detection task, the RPN network is used to predict the feature maps of the top two layers to reduce the number of negative samples and the computational overhead of post-processing NMS to obtain the inspection stamp candidate frames.

[0019] Further, step S3 specifically includes:

[0020] The module receives three feature maps, high-level, medium-level, and low-level, as input. It dynamically upsamples and convolves the high-level feature maps, and resizes the low-level feature maps using depthwise separable convolution and standard convolution. It then divides the three feature maps into four equal parts along the channel dimension, calculates the selective aggregation results using a specific formula, and merges them along the channel dimension to produce the output. This module uses an adjustable fusion factor to complementarily fuse high-level and low-level semantic features, preventing loss of detailed features during cross-level propagation, preventing feature redundancy, and improving the ability to represent fine-grained images.

[0021] Further, step S4 specifically includes:

[0022] Using the complementary multi-level feature maps output by S3, we expand the number of channels through convolution and then perform depthwise separable convolution to separate the keys and values. We calculate the attention score matrix, select the indices with the largest attention scores to set a mask, and perform a softmax operation to obtain a normalized attention matrix. The module uses a dynamic K value to fuse the outputs of different heads, enhancing the multi-level feature relationships and improving fine-grained classification capabilities.

[0023] Further, step S5 specifically includes:

[0024] Combined with the candidate boxes of the inspection stamp generated in S2, the RoIAlign operation is performed on the feature relationship enhanced feature map obtained in S4. The convolution and fully connected layers are used to accurately regress the candidate boxes and fine-grainedly distinguish the inspection stamp categories.

[0025] Another object of the present invention is to provide a method and system for fine-grained detection and recognition of verification stamps based on multi-level feature relationship enhancement, which includes an image preprocessing module, a feature extraction module, a high-quality candidate box generation network module, a complementary feature fusion network module, and a multi-level feature relationship enhancement network module.

[0026] Another object of the present invention is to provide a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the method for fine-grained detection and recognition of verification stamps based on multi-level feature relationship enhancement.

[0027] Another object of the present invention is to provide a verification stamp image data processing terminal, which is used to implement a verification stamp fine-grained detection and recognition system based on a verification stamp fine-grained detection and recognition method enhanced by multi-level feature relationships.

[0028] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0029] First, the present invention presents a fine-grained inspection and recognition method for entry-exit inspection stamps based on enhanced multi-level feature relationships. This method can be used for inspection and location, as well as fine-grained classification, in automated inspection systems for entry-exit inspection stamps. The network innovatively employs a two-stage design architecture. In the first stage, the detection task is cleverly decoupled from the fine-grained classification task, accurately extracting the required information based on the characteristics of each subtask. This is effectively coupled through a carefully designed candidate box mechanism. In the second stage, based on the candidate boxes generated in the first stage, precise regression operations and fine-grained classification are performed on the candidate boxes. The entire model is collaboratively constructed by a detection branch and a classification branch. The detection branch is responsible for outputting the candidate box location information of the inspection stamp and the probability of the inspection stamp being present in the area, providing a basis for subsequent classification. The classification branch focuses on selecting discriminative key features from complex features and performing deep relationship modeling of semantic features at different levels, thereby accurately calculating fine-grained category scores and achieving precise judgment of the inspection stamp category.

[0030] Compared with similar algorithms, the algorithm in this chapter is superior in model size, running speed and technical indicators.

[0031] Second, the present invention proposes a fine-grained verification stamp detection and recognition method based on multi-level feature relationship enhancement, which can provide a theoretical basis and strong technical support for downstream work. For a long time, many passport document inspection devices currently in service have only completed basic passport image acquisition tasks. Further classification of verification stamp types requires manual identification. The present invention proposes a fine-grained verification stamp detection and recognition method based on multi-level feature relationship enhancement. Through multi-level feature relationship enhancement technology, the model of the present invention can keenly capture subtle discriminative features between verification stamp categories, greatly improving the accuracy of verification stamp detection and fine-grained classification capabilities. In practical applications, it can accurately distinguish different types of verification stamps, effectively avoiding the common misclassification problems of traditional methods. It can achieve fine-grained classification of verification stamps from different countries and types, meeting the business requirements for high-precision verification stamp classification and providing reliable guarantees for subsequent verification stamp text information extraction and analysis.

[0032] The fine-grained detection and recognition method for verification stamps based on multi-level feature relationship enhancement proposed in the present invention adopts an end-to-end network structure, which is relatively easy to deploy as a module embedded in the document inspection system, has little impact on the structure of the entire passport collection system, and is easy to deploy; at the same time, since a deep learning framework is adopted to complete the task of fine-grained detection of verification stamps, the module parameters can be trained before being deployed to the system, which makes the time overhead of actually putting this technical solution into production less than that of traditional image processing methods, thereby saving more time for downstream tasks, and enabling the entry and exit passport collection system to complete more personalized, modular collection and reading tasks for the passports to be inspected within a limited time.

[0033] Third, the expected benefits and commercial value of the technical solution of the present invention after transformation are: the expected benefits and commercial value of the transformation of the technical solution are quite considerable.

[0034] In business scenarios such as entry and exit management and customs inspection, traditional verification stamp detection and classification methods rely on manual recognition, which is inefficient and prone to human error. This patented technology can achieve automated, high-precision verification stamp detection and fine-grained classification, greatly improving work efficiency. Taking a busy port as an example, there are many people entering and leaving the country every day. If manual methods are used, a lot of manpower and time are required to verify the verification stamp information. After applying this patented technology, the speed of verification stamp detection and classification is greatly improved, and the inspection of massive documents can be completed quickly and accurately, reducing manual workload and lowering labor costs. It is estimated that in large-scale application scenarios, about 30%-50% of labor costs can be saved.

[0035] Accurate inspection and classification of verification stamps helps identify forged or tampered stamps, effectively preventing security risks such as illegal entry and exit. Traditional methods can struggle to accurately identify attempts to illegally enter or exit the country by forging verification stamps. However, this patented technology, leveraging its powerful multimodal feature fusion and fine-grained classification capabilities, can accurately detect abnormal verification stamps, providing a strong safeguard for border security. This not only reduces security risks associated with illegal entry and exit, but also avoids potential legal risks and economic losses caused by inspection errors, safeguarding national and social security and stability. From a commercial perspective, it can help relevant institutions avoid significant compensation and reputational damage resulting from security incidents.

[0036] This patented technology can be applied in multiple fields. Beyond entry-exit management and customs inspection, it can also be expanded to visa processing agencies, international logistics companies, and more. When reviewing visa applications, visa processing agencies need to confirm the applicant's past entry-exit records. This patented technology can quickly and accurately identify verification stamp information, improving the efficiency and accuracy of visa reviews. International logistics companies handle cross-border cargo transportation, which involves entry-exit declaration and inspection of goods. Accurate identification of verification stamp information helps expedite customs clearance and reduce logistics costs. By collaborating with companies in these fields and transforming patented technologies into actual products or services, substantial commercial benefits can be generated. For example, by collaborating with visa processing agencies, fees can be charged based on the number of services provided or the duration of the collaboration. Customized verification stamp detection systems can be developed for international logistics companies, generating technology licensing fees and subsequent maintenance costs.

[0037] The widespread application of this patented technology will drive the development of the entire image recognition and artificial intelligence industry in related fields. It will stimulate demand for high-performance hardware devices such as image acquisition equipment and computing servers, driving hardware manufacturers to continuously innovate and optimize product performance. It will also drive the continuous optimization and innovation of software algorithms, attracting more R&D resources and creating new growth points for the industry. Through collaboration and collaborative innovation, upstream and downstream companies can jointly develop markets, achieve mutual benefit, and generate more business opportunities and economic benefits for enterprises.

[0038] Fourth, the technical progress of the verification stamp detection method based on multi-level feature relationship enhancement is mainly reflected in the following aspects:

[0039] 1. Improved depth and breadth of feature extraction and fusion: Traditional detection methods can often only extract features of a single scale or limited dimensions of the inspection stamp, making it difficult to fully capture the complex information of the inspection stamp. This method innovatively implements multi-level feature extraction, analyzing the inspection stamp image at different scales and levels, and can obtain rich features from macroscopic contours to microscopic details. For example, through the use of multi-scale convolution kernels, the texture, shape and other features of the inspection stamp are extracted at different resolutions, and then with the help of a multi-level feature relationship enhancement module, these features are effectively fused, making the model's understanding of the inspection stamp more comprehensive and in-depth. This feature extraction and fusion method that combines both depth and breadth greatly improves the model's ability to recognize different types of inspection stamps, and can accurately distinguish even inspection stamps with very similar designs.

[0040] 2. Task decoupling and collaborative optimization: In the past, the simple splitting of the inspection stamp detection and classification tasks easily led to information loss and waste of computing resources. Based on the method of multi-level feature relationship enhancement, the detection and fine-grained classification tasks are cleverly decoupled. In the first stage, information is extracted based on the characteristics of the two tasks respectively, avoiding interference between tasks. At the same time, the collaborative optimization of the two tasks is achieved through a carefully designed candidate box coupling mechanism. For example, after the detection branch outputs the inspection stamp candidate box and the probability of existence, the classification branch can use the discriminative features enhanced by the multi-level feature relationship based on these candidate boxes to perform more accurate fine-grained classification, effectively improving the completion quality and efficiency of the overall task.

[0041] 3. Enhanced model adaptability and generalization: In practical applications, factors such as the image quality, shooting angle, and lighting conditions of the inspection stamp vary widely, making traditional methods difficult to adapt to these complex variations. This method introduces a dynamic adjustment mechanism, such as the application of a dynamic K value in the multi-level feature relationship enhancement module. This method can adaptively adjust the feature fusion strategy and model parameters based on the different inspection stamp image features, thereby enhancing the model's adaptability to different scenarios and conditions. In addition, during training, through a large-scale, diverse dataset and data enhancement techniques, the model learns a rich set of inspection stamp feature patterns, further improving its generalization capabilities. Even when encountering unprecedented inspection stamp types or complex image conditions, it can maintain high detection and classification accuracy.

[0042] 4. Optimization of computational efficiency and performance balance: While pursuing high-precision detection and classification, this method also focuses on improving computational efficiency. Based on the task-decoupled high-quality candidate box generation module, feature maps are targeted for prediction, reducing unnecessary computation. At the same time, in model design, through reasonable network structure and parameter settings, the computational complexity of the model is reduced while ensuring performance. For example, the use of lightweight convolution operations and attention mechanisms significantly improves the model's inference speed without losing too much accuracy, allowing this method to run efficiently with limited hardware resources in practical applications, meeting business scenarios with high real-time requirements.

[0043] The fine-grained detection and recognition method for stamps based on enhanced multi-level feature relationships has achieved significant technical progress in network construction, stamp region detection, discriminative feature discovery for stamps, and fine-grained classification accuracy. These advances not only improve the accuracy and reliability of the task but also provide new insights and more effective technical means for deep learning methods to solve object detection and fine-grained classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0045] Figure 1 This is a structural diagram of the verification stamp automatic inspection system based on the fine-grained detection and recognition method of the verification stamp based on multi-level feature relationship enhancement provided by the embodiment of the present invention.

[0046] Figure 2 This is a flow chart of a fine-grained detection and recognition method for verification stamps based on multi-level feature relationship enhancement provided by an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of a network implementation of a complementary feature fusion module provided by an embodiment of the present invention;

[0048] Figure 4 Schematic diagram of a network implementation of a multi-level feature relationship enhancement module based on cross attention provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0050] In response to the problems existing in the prior art, the present invention provides a fine-grained verification stamp detection method and system based on multi-level feature relationship enhancement. The present invention is described in detail below with reference to the accompanying drawings.

[0051] The following are two specific examples that demonstrate how to apply a fine-grained verification stamp detection method based on multi-level feature relationship enhancement.

[0052] Example 1: Verification stamp detection and fine-grained classification in a real entry-exit inspection environment.

[0053] The hardware device using the method of the present invention is operated in a real entry-exit inspection environment, and images of the verification stamps of licenses and certificates passing through the pilot program are collected daily. Because the device is in a real port environment, there are verification stamp images in a variety of real scenarios, so this embodiment is effective. The entire verification stamp detection and fine-grained recognition process includes:

[0054] 1. Image preprocessing: Input the image that has been preprocessed with size normalization into the entire network.

[0055] 2. Feature extraction and candidate frame generation: This is used to extract image features from the verification stamp and obtain the first-stage target candidate frame.

[0056] 3. Complementary multi-level feature fusion: Combine multi-level feature maps to supplement and fuse the multi-level features in the feature maps.

[0057] 4. Feature relationship enhancement: The correlation between global features and local features is calculated through cross-attention, and the local features are filtered through a masking mechanism to obtain more effective discriminative features for fine-grained classification and suppress other irrelevant features.

[0058] 5. Precise candidate box regression and category classification: The coarse candidate boxes generated by the candidate box generation module are used to perform regional sampling on the fused feature map after feature relationship enhancement, and the results are output through convolution and fully connected layers.

[0059] Example 2: Using passport image samples from a laboratory passport database for verification stamp detection and fine-grained classification. Laboratory passport samples are collected from different image acquisition devices, which can effectively test the generalization performance of the method, thus this example is effective. The entire verification stamp detection and fine-grained recognition process includes:

[0060] 1. Image preprocessing: Input the image that has been preprocessed with size normalization into the entire network.

[0061] 2. Feature extraction and candidate frame generation: This is used to extract image features from the verification stamp and obtain the first-stage target candidate frame.

[0062] 3. Complementary multi-level feature fusion: Combine multi-level feature maps to supplement and fuse the multi-level features in the feature maps.

[0063] 4. Feature relationship enhancement: The correlation between global features and local features is calculated through cross-attention, and the local features are filtered through a masking mechanism to obtain more effective discriminative features for fine-grained classification and suppress other irrelevant features.

[0064] 5. Precise candidate box regression and category classification: The coarse candidate boxes generated by the candidate box generation module are used to perform regional sampling on the fused feature map after feature relationship enhancement, and the results are output through convolution and fully connected layers.

[0065] These two examples demonstrate the wide applicability of the fine-grained stamp detection method based on multi-level feature relationship enhancement in real application scenarios. From a single imaging device in real scenarios to a laboratory environment under acquisition conditions of multiple acquisition devices, this method can effectively improve the accuracy of stamp detection and fine-grained classification, and improve generalization and robustness.

[0066] The detailed image and intermediate data processing process of the fine-grained stamp detection method enhanced by multi-level feature relationships involves multiple steps. The goal is to improve the accuracy of fine-grained stamp classification by learning multi-level complementary features and global-local feature relationships between multi-level features through deep learning methods. The following is a detailed description of each step of the method:

[0067] Step S1: Data input and preprocessing

[0068] Image data preprocessing:

[0069] Resize: The collected passport verification stamp images are uniformly resized and, if necessary, qualitatively cropped to ensure that all input images have the same dimensions for subsequent feature extraction and fusion.

[0070] Instance normalization: Perform instance normalization on the image so that each input sample image has independent normalization parameters, effectively avoiding the mutual influence between training samples.

[0071] Step S2: Image feature extraction and candidate box generation

[0072] Image feature extraction: The backbone network can capture more complex image features. The initial convolutional layer extracts simple low-level features such as edges and textures, and the subsequent residual blocks continuously combine and abstract these features, ultimately forming high-level features that are highly representative of the image content.

[0073] Candidate box generation: Using the top two feature maps in the backbone network, a region proposal network is used to generate candidate regions that may contain the verification stamp.

[0074] Step S3: Complementary multi-level feature fusion

[0075] A feature fusion module is introduced to fuse the features extracted at each stage at multiple scales. This module can adaptively select appropriate features for fusion based on the size and characteristics of the target.

[0076] Step S4: Multi-level feature relationship enhancement

[0077] Based on the cross-attention mechanism, the multi-scale feature map of the backbone network is utilized to enhance the multi-level feature relationship through convolution, attention calculation, and multi-head attention fusion.

[0078] Step S5: Candidate box precise regression and category classification

[0079] Perform precise regression on the candidate boxes generated in S2 and perform fine-grained classification on the candidate box categories to obtain the final result.

[0080] Through the detailed multi-level image feature extraction, fusion, and enhancement process described above, this method can fully capture the subtle differences between different verification stamp categories, improving the ability to separate and extract discriminative features, thereby increasing the accuracy and robustness of the entire network structure for fine-grained classification. In particular, when this method is used in an automated passport verification stamp inspection system, it can effectively perform fine-grained classification of verification stamp types from different countries and categories, significantly reducing the manual workload.

[0081] like Figure 2 As shown, the detailed steps of the fine-grained verification stamp detection method based on multi-level feature relationship enhancement provided by the embodiment of the present invention are as follows:

[0082] Specifically, step S1 involves resizing and instance normalizing the real passport stamp images collected by the passport stamp collection hardware system to ensure that the images input to the network are of consistent size and do not affect each other. Instance normalization refers to the process of calculating the mean and standard deviation of each input feature map on a single channel, and then normalizing the elements within the single channel using the mean and standard deviation on each channel. Instance normalization has significant advantages in image style transfer. For a feature map x with a batch size of N, a length, width, and number of channels of h, w, and c, respectively, its specific mathematical definition is as follows:

[0083]

[0084] where x hw represents an image of scale h×w under a single channel, μ(x) and σ(x) are the mean and variance respectively, γ and β are variables learned during training, and ε is a small constant.

[0085] Specifically, step S2 is to select a backbone network, such as a ResNet network, and construct a deep architecture through convolutional layers, pooling layers, etc. When detecting the verification stamp, the convolution layer convolves the image with convolution kernels of different sizes (such as 3×3, 5×5) to obtain low-level features such as edges and textures. As the network deepens, the residual block uses jump connections to continuously combine and abstract low-level features to solve the gradient disappearance problem, and finally generates highly representative high-level features, providing rich information for subsequent verification stamp detection and classification. The candidate box of the verification stamp is predicted based on the highest two-level feature maps output by the backbone network, thereby generating high-quality candidate areas, reducing the subsequent processing search space, and improving the efficiency of verification stamp detection.

[0086] Specifically, step S3 uses the acquired stamp image feature maps from step 2 as input to a complementary multi-level feature fusion module for subsequent feature reconstruction and output. Specifically, this module considers factors such as the resolution and semantic information richness of feature maps at different levels for complementary fusion.

[0087] Features at different levels contain different semantic information. Shallow features typically contain rich details, such as edges and textures, which are important for detecting small objects and fine structures. Deep features, on the other hand, have stronger semantic expression capabilities and can capture the overall shape and category of an object. Multi-scale feature fusion utilizes feature maps of different scales. Large-scale feature maps capture global information about an object and are suitable for detecting large and distant objects. Small-scale feature maps focus on object details and are helpful for detecting small objects and objects with distinct local features. By fusing these multiple levels of features, we can fully utilize image information and improve the accuracy and robustness of object detection.

[0088] Based on this, a complementary multi-level feature fusion module is proposed. This module can adaptively select appropriate features for fusion based on the size and characteristics of the target. The feature fusion module is introduced to fuse the features extracted at each stage at multiple scales, enabling the network to learn more comprehensive, complementary, and diverse feature information. This module accepts feature maps of high, medium, and low resolutions as input:

[0089]

[0090] For high-level feature maps, dynamic upsampling and a 3×3 convolution operation are performed. For low-level feature maps, depth-wise separable convolution and ordinary convolution operations are performed to adjust the size of the feature maps.

[0091]

[0092] Divide the three feature maps into four equal parts according to the channel dimension, and get: Where h, m, and l represent the high-dimensional, low-dimensional, and i-th partition features of the current layer, respectively. The calculation formula for these partitions is as follows:

[0093] α=sigmoid(m i ),m′ i =αl i +(1-α)h i

[0094]

[0095] in, Indicates m i The value obtained by the activation function, m' i Represents the selective aggregation result of each partition. Merge on the channel dimension After that, we get: Operation Conv(·), and δ(·) represent convolution, batch normalization (BN) and activation function (ReLU), respectively, and the final output is

[0096] For different stamp types, their discriminative features reside in different feature maps of the network. For example, simple texture information requires the network's low-level semantics for judgment, while higher-level feature differences require the introduction of contextual features. The complementary feature fusion module proposed in this section uses an adjustable fusion factor α to complementarily fuse high-level and low-level semantic features. When α > 0.5, the model prioritizes local fine-grained features; when α < 0.5, it prioritizes contextual global features. This complementary multi-level feature fusion module avoids the loss of detailed features during cross-level propagation and, through a complementary mechanism, prevents feature redundancy, improves the representation of fine-grained images, and provides rich semantic features for further discriminative feature screening.

[0097] Step S4 is specifically a multi-level feature relationship enhancement network module. In traditional methods, the discriminative semantic features of multi-level feature maps are not fully exploited, and the key role of the relationship between semantic features and partial features in fine-grained classification is ignored.

[0098] To solve this problem, a multi-level feature relationship enhancement module based on cross attention is proposed.

[0099] This module uses the multi-scale feature map output by the backbone network, denoted as Where H, W, C represent the height, width and number of channels of the feature map respectively, and i∈{1,2,3,4} represents the feature map output at different stages. Among them F x Represents the feature map of each stage, F y Fixed as the feature map of the last stage.

[0100] Use 1×1 convolution to double the number of feature map channels, then process the expanded features through depthwise separable convolution to increase the local receptive field and feature diversity, and then split them according to the channel dimension to obtain keys and values. The formula is shown below.

[0101]

[0102] Calculate the attention score matrix between QK, the output of the standard dot product attention is:

[0103]

[0104] In practical applications, a multi-head attention mechanism is usually used to perform multi-head attention calculations on each new Q, K, and V to obtain an output of d = C / k channel dimensions, and then connect these outputs and obtain the final result through linear projection. Calculate the similarity of pixel pairs between all reshaped queries and keys to obtain a size of The transposed attention matrix M is obtained. In this matrix, the most important k contribution scores are retained through adaptive selection, while the remaining unimportant scores (i.e., elements below the k maximum scores) are masked and their probabilities are set to 0. After the above screening, the sparse attention calculation formula is obtained as follows:

[0105]

[0106] in is a learnable TopK selection operator, which is defined as:

[0107]

[0108] That is, if S ij is the largest value before k in row j, then keep S ij , otherwise it is set to 0. Finally, the result of the SoftMax calculation is matrix multiplied with the value V to obtain the output of the sparse attention. In the multi-head attention mechanism, the outputs of all heads are connected and then linearly projected to obtain the final output.

[0109] For the selective fusion strategy in multi-head attention, a dynamic K value is used in the module to fuse the outputs of different heads. For the K value of each attention head, the weighted weight for each attention head is obtained by weighted calculation of the feature matrix.

[0110] Step S5 is specifically a detection regression subnetwork, which consists of a convolutional layer, a ReLU activation layer, a normalization layer, a fully connected layer, etc. The main function of the detection regression subnetwork is to accurately regress the candidate boxes and complete the fine-grained classification.

[0111] An application embodiment of the present invention provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of a fine-grained verification stamp detection method based on multi-level feature relationship enhancement.

[0112] An application embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of a method for fine-grained detection of a verification stamp based on multi-level feature relationship enhancement.

[0113] An application embodiment of the present invention provides an image data processing terminal, which is used to implement a method and system for fine-grained verification stamp detection based on multi-level feature relationship enhancement.

[0114] The application embodiment of the present invention provides a cross-platform computer application that can be deployed on Windows, Linux, and most domestic operating systems, and is used in conjunction with specific hardware to implement a fine-grained verification stamp detection method and system based on enhanced multi-level feature relationships.

[0115] The present invention can be instantiated through a combination of hardware and software. The hardware implementation mainly relies on the certificate acquisition device. The device uses a natural visible light source to illuminate the certificate placed in the device, and uses a special camera to capture the certificate image. The certificate image acquisition device can also receive relevant instructions from the host computer to control some hardware modules. The host computer software is mainly used to control the specific workflow of the overall system and receive data sent back by the certificate acquisition device. The system communication data content mainly includes certificate image data, camera parameter data, light source parameter data, etc., which also includes the loading of algorithm modules such as the fine-grained detection algorithm of the verification stamp based on multi-level feature relationship enhancement of the present invention. The host computer software processes part of the data through the built-in algorithm and then displays the information to the user in a visual manner on the host computer.

[0116] This paper uses two datasets in total, including a passport verification stamp detection dataset in an actual entry and exit scenario, and a passport verification stamp synthesis dataset.

[0117] This private dataset, created for passport stamp detection in real-world entry and exit scenarios, collects passport stamp images from multiple countries and across different time periods. Using professional image annotation tools, a team of experienced annotators meticulously annotated the stamps in the dataset, including their location, country of origin, and entry / exit type. After annotation, the data was cross-checked by multiple individuals to ensure accuracy and consistency. The resulting dataset contains 2,000 images of various passport stamps, divided into training, validation, and test sets with a 70% / 20% / 10% split. This dataset provides valuable guidance for model training.

[0118] The synthetic passport stamp dataset was constructed to further expand data diversity. Leveraging advanced image synthesis techniques, it simulates a variety of complex real-world scenarios. For example, by varying lighting conditions, including brightness, angle, and shadow effects, it simulates stamp images captured at different times and in different environments. Image clarity is adjusted to generate samples with varying degrees of blur to simulate images acquired under conditions such as variations in acquisition device performance or jitter during the capture process. Furthermore, the color and shape of the stamp are randomly and slightly deformed to simulate wear and fading that can occur over time.

[0119] During the synthesis process, the generated images were ensured to be highly similar in appearance and features to actual verification stamp images, enabling them to reflect the various variations that may occur in real-world scenarios. The verification stamp information in the synthetic dataset was also annotated using professional annotation tools, and a rigorous review process was performed to ensure annotation quality. The resulting synthetic dataset ultimately generated 5,000 verification stamp images, which complement the dataset from actual entry-exit scenarios.

[0120] Combining the two datasets significantly enriches the data sources for model training. The dataset from actual entry-exit scenarios provides the model with authentic, reliable foundational data, enabling it to learn the typical characteristics and patterns of verification stamps in real-world applications. The synthetic dataset further expands the boundaries of the data, exposing the model to a wider range of situations and enhancing its generalization capabilities, enabling it to maintain high detection accuracy and stability in the face of a variety of complex and changing real-world scenarios. During subsequent model training, the training set portions of the two datasets are strategically mixed to allow the model to fully learn the characteristics of data from different sources. The validation and test sets are then used independently to accurately evaluate the model's performance on different types of data, enabling targeted optimization and improvement of the model.

[0121] In order to make full use of high-quality candidate boxes, Adaptive Recognition Loss (ARL) is adopted to mainly solve two problems existing in the current two-stage fine-grained target detection method. First, the standard R-CNN head ignores the quality differences of different candidate boxes. In the fine-grained target detection task, high-quality candidate boxes with higher foreground probability and more precise positioning should be given priority. Secondly, the non-maximum suppression of candidate boxes has a negative impact on fine-grained recognition. Due to the misalignment of classification and positioning in the first stage, high-scoring candidate boxes may not have accurate bounding boxes. Therefore, well-positioned candidate boxes may be removed. The standard Focal Loss can be expressed as:

[0122]

[0123] Where α and γ are two hyperparameters. α is used to balance the contribution of positive and negative samples, while γ adjusts the focus on difficult samples. However, Focal Loss treats positive and negative samples equally, although positive samples are more valuable in fine-grained object detection tasks. In addition, Focal Loss is regulated by cross-entropy, ignoring the actual factors that affect fine-grained recognition. In view of this, a new reweighting strategy is designed. A joint measurement value t is proposed to reweight positive samples according to their quality, calculated as follows (3-7).

[0124]

[0125] Where s represents the proposal classification score provided by RPN, and q is the output IoU after the second-stage box regression, which indicates the quality of the positioning. Overall, ARL is able to jointly evaluate the quality of each proposal based on the first-stage classification and the second-stage positioning. Therefore, ARL can be expressed as:

[0126]

[0127] The hyperparameter β controls the variance of different proposals. Compared to Focal Loss, ARL removes the hyperparameter α because it can also adjust the overall weight of positive samples. By using ARL, it is no longer necessary to perform non-maximum suppression (NMS) on candidate box proposals. Instead, noisy proposals are retained to avoid missing high-quality proposals. Eliminating NMS also significantly improves inference speed. In addition, because ARL can dynamically increase the weight of high-quality proposals, proposals are not sampled during training to maximize the utilization of all high-quality proposals.

[0128] The network architecture was implemented using the Pytorch deep learning framework, employing the Adam optimizer with an initial learning rate of 0.001. During training, an early stopping mechanism was introduced to prevent overfitting. If the model's performance on the validation set did not improve for 10 consecutive epochs, training was stopped and the currently optimized model parameters were saved. Furthermore, a learning rate adjustment strategy was implemented. If the model's performance on the validation set did not decrease for five consecutive epochs, the learning rate was adjusted to 0.1 times the original value, allowing for more precise model convergence in the later stages of training.

[0129] In summary, compared with existing related methods, the method proposed in the present invention is the first algorithm model that combines verification stamp detection and fine-grained classification of verification stamps.

[0130] In summary, compared with existing related methods, the method proposed in this invention is the first end-to-end algorithm model for automated verification of verification stamps that combines detection and fine-grained classification. It is superior in accuracy, model computational complexity, and running time. It is also being tested simultaneously on our company's hardware products, which is conducive to the practical application of this algorithm.

[0131] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes, equivalent substitutions and improvements made by any technician familiar with this technical field within the counting scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A fine-grained verification stamp detection method based on multi-level feature relationship enhancement, characterized in that: The following steps are involved: S1: Image preprocessing: Input the image that has been preprocessed with normalized size into the entire network. S2: Feature extraction and candidate box generation. This is used to extract image features from the verification stamp and obtain the first-stage target candidate box. S3: Complementary multi-level feature fusion. Combine multi-level feature maps to perform complementary fusion of multi-level features in the feature maps. S4: Multi-level feature relationship enhancement. The correlation between global and local features is calculated through cross-attention, and the local features are filtered through a masking mechanism to obtain more effective discriminative features for fine-grained classification and suppress other irrelevant features. S5: Precise candidate box regression and category classification. The coarse candidate boxes generated by the candidate box generation module are used to perform region sampling on the fused feature map after feature relationship enhancement, and the results are output through convolution and fully connected layers.

2. The fine-grained verification stamp detection method with enhanced multi-level feature relationships as claimed in claim 1 is characterized by: Fine-grained stamp detection and recognition is a core component of inspection. However, due to the small differences between different stamp categories and the large number of categories, conventional object detection algorithms struggle to effectively utilize subtle discriminative features, failing to balance detection accuracy with fine-grained classification. To address both stamp detection and fine-grained classification, this method decouples the detection and classification tasks. The detection branch generates high-quality candidate bounding boxes and coarsely locates the stamp region, while the classification branch performs fine-grained feature extraction. The two are coupled through candidate bounding boxes, completing fine-grained classification and boundary regression of the candidate bounding boxes in the second stage of the network.

3. The fine-grained verification stamp detection method based on multi-level feature relationship enhancement as claimed in claim 1 is characterized in that: Step 3 specifically includes: Feature fusion between non-adjacent layers can lead to information loss or degradation. This propagation and interaction process can cause the semantic information of high-level features or the information of low-level features to be lost or degraded. This is especially true in fine-grained classification tasks, where discriminative features often reside in feature maps at lower levels. To address this, a multi-level feature fusion module was designed. The module receives high-, medium-, and low-level feature maps and fuses them by channel partition through operations such as dynamic upsampling and convolution. The fusion factor adaptively selects appropriate features for fusion based on the size and characteristics of the target, avoiding loss of detailed features and redundancy.

4. The fine-grained verification stamp detection method based on multi-level feature relationship enhancement as claimed in claim 1 is characterized in that: Step 4 specifically includes: Traditional methods fail to fully exploit the semantic features of multi-level feature maps and overlook the crucial role of the relationship between semantic features and partial features in fine-grained classification. To address this issue, a multi-level feature relationship enhancement module based on cross-attention is proposed. The cross-attention mechanism is based on the multi-scale feature maps output by the backbone network. It uses the feature maps output at different stages as queries, keys, and values, respectively, and calculates the similarity between the query and the key to obtain attention weights. This process enables the model to deeply analyze the degree of correlation between feature maps at different levels, thereby uncovering hidden relationships between multi-level features.

5. The fine-grained verification stamp detection method based on multi-level feature relationship enhancement as claimed in claim 1 is characterized in that: Step 4 specifically includes: The traditional attention mechanism's "global aggregation, global application" model easily obscures the discriminative information of the stamp of approval by the background. However, the cross-attention mechanism, after calculating the attention score matrix, selects the indices with the largest attention scores and masks these indexes, excluding them from the calculation. This approach allows the model to focus on small, discriminative regions and features in the image, avoiding interference from noisy backgrounds and fully utilizing the discriminative component features, thereby improving the accuracy of fine-grained classification of stamps of approval.

6. A system for implementing the fine-grained verification stamp detection method based on multi-level feature relationship enhancement as described in any one of claims 1 to 5, characterized in that: It includes complementary multi-level feature fusion network module, multi-level feature relationship enhancement network module, and fine-grained classification regression sub-network.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the fine-grained verification stamp detection method based on multi-level feature relationship enhancement as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the fine-grained verification stamp detection method based on multi-level feature relationship enhancement as described in any one of claims 1 to 6.

9. A passport verification stamp image data processing terminal, the terminal being used to implement the passport verification stamp image automated inspection system of the fine-grained verification stamp detection method based on multi-level feature relationship enhancement as claimed in claim 6.

Images