An attention guide-based multi-stage feature fusion seal recognition and character extraction method, device and medium

Abstract

The application discloses a kind of based on attention guide's multi-stage feature fusion seal identification and character extraction method, device, medium and equipment, the method includes, the picture containing seal is input improved YOLOv10 model, extracts the seal area in the picture, wherein the improved YOLOv10 model is the new model formed after adding attention guide module after the feature extraction module of YOLOv10 main network;The seal area is input seal character extraction model, and the character in the seal area is extracted.The application improves the identification effectiveness of seal area in picture in complex scene and the extraction accuracy of character in seal.

Description

Technical Field

[0001] This invention relates to the fields of target detection and computer vision technology, and in particular to a method, apparatus and medium for seal recognition and text extraction based on attention-guided multi-stage feature fusion. Background Technology

[0002] With rapid economic growth and increasingly frequent transactions, government and business transactions generate a large number of contracts, invoices, and other documents every day. The legality, validity, and authenticity of these documents require rigorous verification. However, traditional manual review methods not only consume a lot of human resources but also struggle to meet the demands for efficient and accurate verification.

[0003] In recent years, with the development of machine learning and deep learning technologies, computer vision-based seal detection and text recognition methods have been increasingly applied to document review scenarios. In seal detection, existing technologies typically rely on image color features or edge features for candidate region extraction. For example, some schemes are based on the YCbCr (Luminance-BlueChrominance-Red Chrominance) color space, using projection analysis of chromaticity components to locate the seal area. This type of scheme has some effectiveness in color seal detection scenarios, but its applicability is poor in grayscale images or black seal scenarios. Furthermore, researchers have proposed seal detection methods that do not rely on color information. These methods typically extract connected regions through image scaling and edge detection, filter candidate regions based on rules, and finally locate the seal through geometric shape verification. However, these methods are generally only suitable for regular shapes, such as circular or elliptical seals, and their accuracy is low in complex backgrounds or degraded seal scenarios. Summary of the Invention

[0004] The purpose of this invention is to propose a method, apparatus, and medium for seal recognition and text extraction based on attention-guided multi-stage feature fusion. By introducing an attention-guided mechanism after the feature extraction module of the YOLOv10 backbone network, and combining it with a sequence decoding structure based on ResNet-50 multi-stage feature reuse coding and a combination of a transformer network and a bidirectional long short-term memory network, the effectiveness of seal region recognition in images and the accuracy of text extraction from seals in complex scenes are improved.

[0005] To achieve the above objectives, a first aspect of the present invention provides a method for seal recognition and text extraction based on attention-guided multi-stage feature fusion, the method comprising: An image containing a seal is input into an improved YOLOv10 model to extract the seal region from the image. The improved YOLOv10 model is a new model formed by adding an attention guidance module after the feature extraction module of the YOLOv10 backbone network. Input the seal area into the seal text extraction model to extract the text in the seal area.

[0006] In this embodiment, by accurately locating the seal area before extracting the text during the detection stage, the interference of non-seal background on recognition is reduced, thereby improving the accuracy of seal text extraction.

[0007] Optionally, the stamp area in the image can be extracted through the following steps: The first feature matrix output by the feature extraction module is compressed to obtain several attention matrices, and the several attention matrices are fused to obtain the first attention weight matrix; Perform matrix multiplication between the first attention weight matrix and the first feature matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed, and a second attention weight vector is generated through a weight generation unit; The second attention weight vector is activated and then multiplied with the first enhanced feature matrix to obtain the attention enhanced feature matrix. The stamp region in the image is extracted based on the attention-enhanced feature matrix.

[0008] In this embodiment, an attention-enhanced feature matrix is ​​generated through two-level attention weighting, which improves the accuracy of seal recognition in complex backgrounds and dark-colored scenes, and reduces the probability of misidentifying background textures, text, and other noise as seals.

[0009] Optionally, the first feature matrix output by the feature extraction module is compressed to obtain several attention matrices, and the several attention matrices are fused to obtain a first attention weight matrix, including: The first feature matrix output by the feature extraction module is compressed using the global average pooling layer and the global max pooling layer in the attention guidance module, respectively, to obtain the first attention matrix and the second attention matrix. The first attention matrix and the second attention matrix are concatenated, and a convolution operation is performed on the concatenated matrix to obtain an intermediate feature matrix; The intermediate feature matrix is ​​subjected to Softmax activation to obtain the first attention weight matrix.

[0010] In this embodiment, complementary statistical information from global average pooling and global max pooling is used to obtain overall distribution features and significant response features. Then, weight distribution is obtained through convolutional fusion and Softmax normalization. This can more accurately highlight key information such as seal texture and edges, reduce the weight bias caused by relying solely on a single pooling method, and improve the accuracy of seal feature recognition.

[0011] Optionally, the step of performing feature compression on the first enhanced feature matrix and generating a second attention weight vector through a weight generation unit includes: Perform matrix multiplication on the first feature matrix and the first attention weight matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed using a global max pooling layer to obtain the third attention matrix; The third attention matrix is ​​processed using a fully connected layer to obtain the second attention weight vector.

[0012] In this embodiment, by compressing the enhanced features and generating weight vectors by the fully connected layer, the key spatial regions are re-selected and re-weighted, further suppressing the response of noise in the outer region of the seal, such as text, noise, and background, so that the final attention enhancement features are more focused on the main body region of the seal, thereby improving the seal boundary positioning accuracy.

[0013] Optionally, the step of inputting the seal area into the seal text extraction model and extracting the text in the seal area includes: The stamp region is input into a feature multiplexing encoder to extract the enhanced fusion features of the stamp region; The enhanced fusion features are input into the feature decoder to extract the text in the stamp area.

[0014] In this embodiment, text extraction is divided into two stages: enhanced fusion feature construction and sequence feature decoding. This allows the encoding end to focus on generating semantically readable features, while the decoding end focuses on sequence modeling and character output. This reduces the reliance of end-to-end recognition on a single feature layer and improves the recoverability in cases of missing character strokes or worn seals, thereby enhancing the accuracy of seal character recognition.

[0015] Optionally, if the feature multiplexing encoder is a multi-stage feature extraction network built on ResNet-50, then the enhanced fusion features of the stamp region are extracted using the feature multiplexing encoder through the following steps: The feature multiplexing encoder is used to extract features from the stamp area to obtain first image features, and convolution operation is performed on the first image features to obtain the first image features after dimensionality reduction. Feature extraction is performed on the first image features to obtain the second image features, and convolution operation is performed on the second image features to obtain the dimension-reduced second image features; The second image features are extracted to obtain the third image features, and the third image features are convolved to obtain the dimension-reduced third image features. The third image features are extracted to obtain the fourth image features; The first image feature, the second image feature, the third image feature, and the fourth image feature after dimensionality reduction are concatenated, and the concatenated feature data is convolved to obtain the enhanced fusion feature.

[0016] In this embodiment, by reusing and fusing multi-stage features, fine-grained information such as the strokes and edges of the characters on the seal is preserved. Furthermore, dimensionality reduction can reduce redundant channels and computational load. After splicing, convolution is performed to achieve cross-layer information integration, resulting in enhanced fusion features that are more suitable for character recognition, thereby improving the character recognition in blurry, incomplete, or unevenly colored seals.

[0017] Optionally, the step of inputting the enhanced fusion features into the feature decoder to extract the text in the stamp region includes: The enhanced fusion features are serialized to obtain a feature sequence; The feature sequence is input into the transformer network module, and the feature sequence is processed globally through a self-attention mechanism to obtain a context-enhanced feature sequence. The context-enhanced feature sequence is input into a bidirectional long short-term memory network, and the context-enhanced feature sequence is processed in the temporal dimension to obtain character prediction features corresponding to each temporal position. The character prediction features are input into the temporal classification layer to calculate the character probability distribution corresponding to each temporal position. Sequence decoding is performed based on the character probability distribution to obtain the text in the seal area.

[0018] In this embodiment, a transformer network is used to model the global relationship of the sequence; then, a bidirectional long short-term memory network is used to model the sequence in both directions in the temporal dimension, which enhances the continuity constraint of adjacent characters; the sequence can be output without character-by-character alignment, which reduces the recognition deviation caused by labeling and alignment errors, thereby improving the recognition accuracy of irregularly arranged seal characters.

[0019] Optionally, the connection-time classification layer uses the negative log-likelihood of conditional probability as a loss function and optimizes the parameters of the transformer network module and the bidirectional long short-term memory network through backpropagation.

[0020] In this embodiment, the negative log-likelihood of the sequential classification layer is used as the training objective. The parameters of the transformer network and the bidirectional long short-term memory network are jointly optimized through backpropagation to improve the consistency of feature modeling and sequence decoding, thereby enhancing the training convergence stability and final recognition accuracy.

[0021] To achieve the above objectives, a second aspect of the present invention also provides an attention-guided multi-stage feature fusion stamp recognition and text extraction device, used to implement the attention-guided multi-stage feature fusion stamp recognition and text extraction device described in any of the first aspects above, the device comprising: The stamp region extraction module is used to input an image containing a stamp into an improved YOLOv10 model and extract the stamp region in the image. The improved YOLOv10 model is a new model formed by adding an attention guidance module after the feature extraction module of the YOLOv10 backbone network. The seal text extraction module is used to input the seal area into the seal text extraction model and extract the text in the seal area.

[0022] A third aspect of the present invention also provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the attention-guided multi-stage feature fusion seal recognition and text extraction method described in any of the first aspects above. Attached Figure Description

[0023] Figure 1 This is a flowchart of a preferred embodiment of a multi-stage feature fusion stamp recognition and text extraction method based on attention guidance provided in the first aspect of the present invention; Figure 2 This is a schematic diagram of an attention-guided YOLOv10 model, representing another preferred embodiment of the attention-guided multi-stage feature fusion stamp recognition and text extraction method provided in the first aspect of the present invention. Figure 3 This is a schematic diagram of the attention guidance module of another preferred embodiment of the attention-guided multi-stage feature fusion seal recognition and text extraction method provided in the first aspect of the present invention; Figure 4 This is a schematic diagram of a seal text extraction module, which is another preferred embodiment of the attention-guided multi-stage feature fusion seal recognition and text extraction method provided in the first aspect of the present invention. Figure 5 This is a structural block diagram of a preferred embodiment of an attention-guided multi-stage feature fusion stamp recognition and text extraction device provided in the second aspect of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that the data involved in this invention (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0026] In this embodiment of the invention, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplarily" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0027] In this invention description, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In this invention description, unless otherwise stated, "a plurality of" means two or more. In this invention description, the term "comprising" and its variations are open-ended, meaning "including but not limited to." The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments."

[0028] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0029] In the description of this invention, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0030] The following explains some concepts involved in the embodiments of the present invention: The first aspect of this invention provides a method for stamp recognition and text extraction based on attention-guided multi-stage feature fusion, see [link to relevant documentation]. Figure 1 The diagram shown is a flowchart of a preferred embodiment of a multi-stage feature fusion stamp recognition and text extraction method based on attention guidance provided in the first aspect of the present invention. The method includes steps S1 to S2, as follows: Step S1: Input the image containing the seal into the improved YOLOv10 model and extract the seal area in the image. The improved YOLOv10 model is a new model formed by adding an attention guidance module after the feature extraction module of the YOLOv10 backbone network. In one example, a terminal device or server acquires a scanned image of a contract to be processed. This image can come from a scanner, a mobile camera, a historical archive, or an upload interface from a business system. The image is then input into a seal region extraction module. This module uses an improved YOLOv10 model to extract features from the image. During feature extraction, an attention-guided module enhances key features, outputting the location results of candidate seal regions. Further, the seal region extraction module encapsulates the location results and sends them to a subsequent seal text extraction module.

[0031] It should be noted that the positioning results may include, but are not limited to, the bounding box coordinates of the stamp area, confidence information, or a cropped image of the stamp area.

[0032] In this embodiment, by introducing an attention-guided mechanism in the feature extraction stage, the model can still highlight the seal-related feature responses even in complex backgrounds, when the seal color is dark or similar to the color of the text, thereby improving the accuracy and robustness of seal area localization and reducing the impact of missed detections and false detections on subsequent text recognition.

[0033] In a specific implementation, see Figure 2This is a schematic diagram of an attention-guided YOLOv10 model, representing another preferred embodiment of the attention-guided multi-stage feature fusion method for seal recognition and text extraction provided in the first aspect of this invention. The improved YOLOv10 model generally includes a feature extraction stage, an intermediate feature fusion stage, and a detection stage. The input image first undergoes multi-layer feature extraction through multiple convolutional layers (Conv) and a cross-stage partial with two convolutions (C2f). After multiple feature extraction nodes in the backbone network, an attention module (Attention) is inserted to weight and enhance the feature map, thereby highlighting key features related to the seal and suppressing background noise. In the feature extraction stage, the model extracts shallow and deep features layer by layer through the Conv and C2f modules, and adds an Attention module after the key layer for feature recalibration. Subsequently, it further integrates multi-scale information through a Spatial-Channel Decoupled Downsampling Module (SCDown) and an Inverted Bottleneck (C2fCIB) feature extraction module. Furthermore, it enhances the ability to express multi-scale contextual information through a Spatial Pyramid Pooling Fast (SPPF) module and a Partial Self-Attention Module (PSA). Upsampling and concatenation operations (Concat) are used to fuse features at different scales. In the detection stage, the fused features are input to the v10 Detection Head (v10Detect) for target localization and category prediction, finally outputting the detection results for the stamp region.

[0034] Step S2: Input the seal area into the seal text extraction model to extract the text in the seal area.

[0035] In one example, the seal text extraction module receives seal data from step S1 and extracts a seal region image from the original image based on the data. Subsequently, the seal region image is input into the seal text extraction model for text extraction. The seal text extraction model may include two stages: feature encoding and feature decoding. In the encoding stage, features are extracted from the seal region to generate enhanced fusion features. In the decoding stage, sequence modeling and character decoding are performed on the enhanced fusion features to output the text result corresponding to the seal region. Furthermore, the seal text extraction module can associate the text result with the seal region location result to form a structured output, which is then returned to the upper-level business system for contract verification, archiving, and other processing.

[0036] It should be noted that the text results may include, but are not limited to, text content, character sequence confidence, and text position mapping information.

[0037] In this embodiment, targeted text extraction from the Region of Interest (ROI) of the seal avoids background interference and misidentification caused by indiscriminate recognition of the entire document. At the same time, by fusing and modeling the features of the seal area and performing sequence decoding, the recognizability of the text in the seal is improved in cases of blurriness, missing parts, edge wear, or superimposed noise, thereby improving the accuracy and stability of text extraction.

[0038] In another preferred embodiment, the stamp area in the image is extracted through the following steps: The first feature matrix output by the feature extraction module is compressed to obtain several attention matrices, and the several attention matrices are fused to obtain the first attention weight matrix; Perform matrix multiplication between the first attention weight matrix and the first feature matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed, and a second attention weight vector is generated through a weight generation unit; The second attention weight vector is activated and then multiplied with the first enhanced feature matrix to obtain the attention enhanced feature matrix. The stamp region in the image is extracted based on the attention-enhanced feature matrix.

[0039] In one example, when the terminal device or server executes step S1, it inputs the image to be detected into the stamp region extraction module. The stamp region extraction module calls the feature extraction module to extract features from the image, obtaining a first feature matrix F. Subsequently, the attention weight generation module performs feature compression from the first feature matrix F to obtain multiple attention matrices, for example, representing attention responses of different statistics or different dimensions, and generates a first attention weight matrix W1 through fusion processing. The stamp region extraction module performs matrix multiplication between W1 and the first feature matrix F to obtain a first enhanced feature matrix F1.

[0040] Furthermore, the attention weight generation module performs feature compression on the first enhanced feature matrix F1 and generates a second attention weight vector W2 through the weight generation unit. Subsequently, W2 is activated to obtain the second attention weight that can be used for weighting; the stamp region extraction module performs matrix multiplication on the second attention weight and the first enhanced feature matrix F1 to obtain the attention enhanced feature matrix F2.

[0041] Finally, the stamp region extraction module inputs the attention enhancement feature matrix F2 into the detection head / candidate region generation unit and outputs the stamp region localization result. The localization result may include bounding box coordinates, confidence score and a set of optional candidate regions, and the localization result is sent as ROI data to the subsequent text extraction process.

[0042] It should be noted that the weight generation unit may include, but is not limited to, fully connected maps, convolutional maps, attention maps, or combinations thereof; the activation function may include, but is not limited to, Softmax, Sigmoid, or a normalization function. In this embodiment, by generating two levels of attention weights and performing two weighted enhancements, the accuracy and stability of seal area positioning are improved and the probability of false detection and false negative detection is reduced, even when the seal color is dark, the background noise is complex, or the seal overlaps with the text.

[0043] In another preferred embodiment, the first feature matrix output by the feature extraction module is compressed to obtain several attention matrices, and the several attention matrices are fused to obtain a first attention weight matrix, including: The first feature matrix output by the feature extraction module is compressed using the global average pooling layer and the global max pooling layer in the attention guidance module, respectively, to obtain the first attention matrix and the second attention matrix. The first attention matrix and the second attention matrix are concatenated, and a convolution operation is performed on the concatenated matrix to obtain an intermediate feature matrix; The intermediate feature matrix is ​​subjected to Softmax activation to obtain the first attention weight matrix.

[0044] In one example, see Figure 3 The diagram shown is a schematic of the attention-guided module in another preferred embodiment of the attention-guided multi-stage feature fusion stamp recognition and text extraction method provided in the first aspect of the present invention. After feature extraction is completed by the C2f module in the backbone network of the improved YOLOv10, it outputs the first feature matrix. Subsequently, the attention guidance module... Two compression branches are executed in parallel. Attention matrices are obtained along the channel dimension using both global average pooling and global max pooling layers. , Furthermore, the Concat function is used to combine the first attention matrix... Second attention matrix The matrices are concatenated, and then convolution and Softmax activation are applied to the concatenated matrix. Finally, the output of Softmax is compared with the first feature matrix. Perform matrix multiplication to generate the first enhanced feature matrix after channel attention. The formulas corresponding to the above content are shown below:

[0045]

[0046]

[0047] In this embodiment, complementary attention information is obtained through dual-branch compression, and then fused by convolution and normalized by Softmax. This approach can simultaneously take into account global statistics and local saliency, making the generated first attention weight matrix more sensitive to seal-related responses. This improves the accuracy of seal feature recognition under conditions such as complex backgrounds, dark seal colors, or seals that are similar to the main text.

[0048] In yet another preferred embodiment, the step of performing feature compression on the first enhanced feature matrix and generating a second attention weight vector through a weight generation unit includes: Perform matrix multiplication on the first feature matrix and the first attention weight matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed using a global max pooling layer to obtain the third attention matrix; The third attention matrix is ​​processed using a fully connected layer to obtain the second attention weight vector.

[0049] In one example, for the generated first enhanced feature matrix Further self-attention is performed in the spatial dimension. Specifically, firstly, for features... A global max pooling layer is used to obtain the second attention weight vector in the spatial dimension. Then, a fully connected layer (FC) and softmax are used to process the second attention weight vector. Finally, the output of softmax is combined with the first enhanced feature matrix. Perform matrix multiplication to generate the attention-enhanced feature matrix after spatial attention. .

[0050] In this embodiment, based on the first stage of attention enhancement, a second attention weight vector is generated through max pooling compression and fully connected mapping mechanism to achieve secondary screening and enhancement of salient features. This can effectively suppress high-frequency interference responses such as the background, text, and background patterns outside the seal, making the attention enhancement features more concentrated in the main area of ​​the seal, reducing false detections and improving the accuracy of seal boundary positioning.

[0051] In another preferred embodiment, the step of inputting the seal area into the seal text extraction model and extracting the text in the seal area includes: The stamp region is input into a feature multiplexing encoder to extract the enhanced fusion features of the stamp region; The enhanced fusion features are input into the feature decoder to extract the text in the stamp area.

[0052] In one example, when the terminal device or server performs step S2 above, the text extraction module receives ROI data from the stamp region extraction module and extracts a stamp region image from the original image based on the ROI data, or directly receives a cropped image of the stamp region as the stamp ROI. Subsequently, the text extraction module inputs the stamp ROI into the feature multiplexing encoder to generate enhanced fusion features.

[0053] It should be noted that the feature reuse encoder can be a multi-stage feature extraction network, which may include, but is not limited to, residual networks, convolutional networks, lightweight networks or combinations thereof, and can output features from different stages and reuse and fuse them to obtain enhanced fused features that simultaneously contain semantic information.

[0054] Furthermore, the text extraction module inputs the enhanced fusion features into the feature decoder for text extraction. The feature decoder may include, but is not limited to, attention-based decoding structures, sequence modeling structures, or end-to-end text recognition structures. For example, it may employ a decoding unit based on a self-attention mechanism, a recurrent neural network decoding unit, a Connectionist Temporal Classification (CTC) decoding unit, or a combination thereof, to output recognition results in the form of character sequences. The text extraction module may further output confidence information associated with the recognition results, or establish a mapping relationship between the text results and the ROI position of the seal, forming a structured output and returning it to the business system.

[0055] In this embodiment, targeted text extraction of the seal area is performed through feature encoding and decoding, reducing the interference of the background of the whole page document, table lines or body text on the recognition. At the same time, the enhanced fusion features output by the feature reuse encoder take into account high-level semantic information, so that the character recognition can still be maintained even when the seal is worn, blurred, missing or superimposed with noise, thereby improving the accuracy and robustness of seal text extraction.

[0056] In another preferred embodiment, the feature reuse encoder is a multi-stage feature extraction network built on ResNet-50. The enhanced fusion features of the stamp region are extracted using the feature reuse encoder through the following steps: The feature multiplexing encoder is used to extract features from the stamp area to obtain first image features, and convolution operation is performed on the first image features to obtain the first image features after dimensionality reduction. Feature extraction is performed on the first image features to obtain the second image features, and convolution operation is performed on the second image features to obtain the dimension-reduced second image features; The second image features are extracted to obtain the third image features, and the third image features are convolved to obtain the dimension-reduced third image features. The third image features are extracted to obtain the fourth image features; The first image feature, the second image feature, the third image feature, and the fourth image feature after dimensionality reduction are concatenated, and the concatenated feature data is convolved to obtain the enhanced fusion feature.

[0057] In one example, see, Figure 4 The diagram shown is a schematic diagram of the seal text extraction module of another preferred embodiment of the attention-guided multi-stage feature fusion seal recognition and text extraction method provided in the first aspect of the present invention.

[0058] The seal text extraction model first crops and aligns the Region of Interest (ROI) to a fixed input size and then feeds it into a feature multiplexing encoder. The feature multiplexing encoder uses a multi-stage ResNet-50 structure for step-by-step feature extraction, outputting image features in four stages, the first image feature... Second image features Third image features and fourth image features Then, regarding , , Perform convolutional dimensionality reduction processing separately to obtain the first image features after dimensionality reduction. , , The dimensionality reduction can be achieved using convolution operations to reduce the number of channels and minimize redundant features. For example, the formula for dimensionality reduction is as follows:

[0059] Next, , , With the fourth image feature The data is concatenated according to a preset dimension, and then convolutional operations are performed on the concatenated feature data to output enhanced fused features. :

[0060] In this embodiment, low-level texture details such as strokes and edges are fused with high-level semantic structures through ResNet-50 multi-stage feature reuse; and redundant channels are suppressed by dimensionality reduction and noise propagation are reduced, making the resulting enhanced fusion features more suitable for the restoration and recognition of characters in seals, and improving the recognizability in cases of blurriness, wear or uneven ink color.

[0061] In yet another preferred embodiment, the step of inputting the enhanced fusion feature into the feature decoder to extract the text in the stamp region includes: The enhanced fusion features are serialized to obtain a feature sequence; The feature sequence is input into the transformer network module, and the feature sequence is processed globally through a self-attention mechanism to obtain a context-enhanced feature sequence. The context-enhanced feature sequence is input into a bidirectional long short-term memory network, and the context-enhanced feature sequence is processed in the temporal dimension to obtain character prediction features corresponding to each temporal position. The character prediction features are input into the temporal classification layer to calculate the character probability distribution corresponding to each temporal position. Sequence decoding is performed based on the character probability distribution to obtain the text in the seal area.

[0062] In one example, see Figure 4 As shown. The feature decoder receives the enhanced fused features. Then, firstly, the serialization module will... The feature sequence is expanded according to preset rules. For example, it is expanded column by column along the width, with each column's feature vector serving as a time step, thus mapping the two-dimensional features into an input sequence for adaptive sequence modeling. Subsequently, the feature sequence is input into the transformer network module. The transformer network uses a self-attention mechanism to calculate the correlation between positions within the sequence, performing global modeling of the feature sequence and outputting a context-enhanced feature sequence. This process can better capture cross-positional correlations caused by circular arrangements or inter-character interference in seal text.

[0063] Furthermore, the context-enhanced feature sequence is input into a Bidirectional Long Short-Term Memory Network (BiLSTM). BiLSTM extracts contextual information in both the forward and reverse temporal directions, outputting character prediction features for each temporal position. The input is then connected to a temporal classification layer to calculate the character probability distribution for each temporal position. Finally, sequence decoding is performed based on the character probability distribution to output the text result in the stamp area.

[0064] It is understood that the decoding can employ greedy decoding or bundle search decoding, and post-processing such as removing whitespace characters and merging duplicate characters can be performed to obtain the final character sequence.

[0065] In this embodiment, a transformer network provides global dependency modeling capabilities, BiLSTM provides temporal continuity constraints, and the temporal classification layer outputs time-series character probabilities, making the recognition process more tolerant of irregular arrangements, character adhesion, or partial defects, thereby improving the accuracy and stability of seal character extraction.

[0066] In another preferred embodiment, the connection-time classification layer uses the negative log-likelihood of conditional probability as a loss function and optimizes the parameters of the transformer network module and the bidirectional long short-term memory network through backpropagation.

[0067] In one example, during the model training phase, the system acquires training sample pairs including the stamp ROI image and its corresponding text annotation sequence. For each training sample, forward inference is performed to obtain the character probability distribution at each temporal position. The temporal classification layer uses the CTC training criterion, mapping the character probability distribution to the target label sequence. This calculates the probability of the label sequence without requiring precise character-by-character alignment of the annotations, and uses the negative log-likelihood of the conditional probability as the loss function. Subsequently, the training module is based on the loss function. The gradient is calculated through backpropagation, and the parameters of the transformer network module and BiLSTM are iteratively updated.

[0068] It should be noted that gradients can also be backpropagated to jointly optimize the parameters of the feature reuse encoder. The optimizer can employ stochastic gradient descent, Adam, or other methods, and the learning rate, number of iterations, and batch size can be configured according to the task scale.

[0069] In this embodiment, by jointly optimizing the transformer network and BiLSTM parameters through backpropagation, the consistency between global attention modeling and temporal modeling can be enhanced, thereby improving the training convergence stability and final accuracy of seal character recognition.

[0070] In another preferred embodiment, to verify the effectiveness of the seal detection and text extraction method proposed in this invention, a publicly available seal dataset is used to train and test the model. The dataset originates from the PaddlePaddle AI platform; the data annotation file for the seal detection task is in VOC format, and the annotation file for the text recognition task is in TXT format, used to record the corresponding character sequence information.

[0071] In seal detection tasks, the Intersection over Union (IOU), Precision (Pre), and Recall (Rec) are used as evaluation metrics, and their calculation formulas are as follows:

[0072]

[0073]

[0074] Where IOU represents the intersection-union ratio between the predicted and ground truth bounding boxes; Pre represents the proportion of predicted positive samples that are actually positive samples; and Rec represents the proportion of actually positive samples that are correctly predicted as positive samples. The predicted detection bounding box region for seal detection is represented by the ground truth value denoted as . .

[0075] In text recognition tasks, text recognition accuracy is used as an evaluation metric to measure the consistency between the recognition results and the actual labeled sequences.

[0076] The proposed seal detection and character recognition model is implemented based on the PyTorch deep learning framework. The experimental training environment was a computing platform equipped with two NVIDIA RTX3090 graphics processing units with 24GB of video memory. In the character recognition task, due to the large number of Chinese character categories, to ensure sufficient model convergence, the number of training iterations was set to 400,000, the initial learning rate was set to 0.0001, and the total training time was approximately 10 hours.

[0077] Table 1. Experimental Results of Seal Detection

[0078] In the seal detection task, the method proposed in this invention is compared with several commonly used object detection algorithms, including Fast R-CNN, YOLO, and YOLOv10. The experimental results are shown in Table 1. The method of this invention achieves superior results in IoU, Pre, and Rec metrics, with IoU reaching 95.32%, Pre reaching 96.75%, and Rec reaching 97.28%. This result demonstrates that by introducing an attention-guided mechanism, the model exhibits higher accuracy and stability in seal region localization.

[0079] Table 2 Comparison Results of Text Recognition Experiments

[0080] In the character recognition task, the method of this invention was compared with common text recognition models such as CRNN, RARE, and MA-CRNN. ​​The experimental results are shown in Table 2. The character recognition accuracy of the method of this invention reached 96.35%, which is higher than the comparison models. This result indicates that the recognition accuracy of characters in seals can be effectively improved through the collaborative design of multi-stage feature fusion and sequence modeling structures.

[0081] A second aspect of this invention provides an attention-guided multi-stage feature fusion stamp recognition and text extraction device, used to implement the attention-guided multi-stage feature fusion stamp recognition and text extraction method described in any of the first aspects above. See also... Figure 5 The diagram shown is a structural block diagram of a preferred embodiment of an attention-guided multi-stage feature fusion stamp recognition and text extraction device provided in the second aspect of the present invention. The device includes: The stamp region extraction module 11 is used to input an image containing a stamp into an improved YOLOv10 model and extract the stamp region in the image. The improved YOLOv10 model is a new model formed by adding an attention guidance module after the feature extraction module of the YOLOv10 backbone network. The seal text extraction module 12 is used to input the seal area into the seal text extraction model and extract the text in the seal area.

[0082] Preferably, the seal area extraction module 11 specifically includes: The first stamp region extraction unit is used to compress the first feature matrix output by the feature extraction module to obtain several attention matrices, and to fuse the several attention matrices to obtain the first attention weight matrix. The second stamp region extraction unit is used to perform matrix multiplication on the first attention weight matrix and the first feature matrix to obtain the first enhanced feature matrix. The third stamp region extraction unit is used to compress the features of the first enhanced feature matrix and generate a second attention weight vector through the weight generation unit. The fourth stamp region extraction unit is used to activate the second attention weight vector and perform matrix multiplication with the first enhanced feature matrix to obtain the attention enhanced feature matrix. The fifth stamp region extraction unit is used to extract the stamp region in the image based on the attention enhancement feature matrix.

[0083] Preferably, the first stamp region extraction unit is used to extract the first attention weight matrix through the following steps: The first feature matrix output by the feature extraction module is compressed using the global average pooling layer and the global max pooling layer in the attention guidance module, respectively, to obtain the first attention matrix and the second attention matrix. The first attention matrix and the second attention matrix are concatenated, and a convolution operation is performed on the concatenated matrix to obtain an intermediate feature matrix; The intermediate feature matrix is ​​subjected to Softmax activation to obtain the first attention weight matrix.

[0084] Preferably, the third stamp region extraction unit is used to obtain the second attention weight vector through the following steps: Perform matrix multiplication on the first feature matrix and the first attention weight matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed using a global max pooling layer to obtain the third attention matrix; The third attention matrix is ​​processed using a fully connected layer to obtain the second attention weight vector.

[0085] Preferably, the seal text extraction module 12 specifically includes: The first seal text extraction unit is used to input the seal area into the feature multiplexing encoder and extract the enhanced fusion features of the seal area; The second seal text extraction unit is used to input the enhanced fusion features into the feature decoder to extract the text in the seal area.

[0086] Preferably, the first seal text extraction unit is used to obtain the enhanced fusion feature through the following steps: The feature multiplexing encoder is used to extract features from the stamp area to obtain first image features, and convolution operation is performed on the first image features to obtain the first image features after dimensionality reduction. Feature extraction is performed on the first image features to obtain the second image features, and convolution operation is performed on the second image features to obtain the dimension-reduced second image features; The second image features are extracted to obtain the third image features, and the third image features are convolved to obtain the dimension-reduced third image features. The third image features are extracted to obtain the fourth image features; The first image feature, the second image feature, the third image feature, and the fourth image feature after dimensionality reduction are concatenated, and the concatenated feature data is convolved to obtain the enhanced fusion feature.

[0087] Preferably, the second seal text extraction unit is used to extract the text in the seal area through the following steps: The enhanced fusion features are serialized to obtain a feature sequence; The feature sequence is input into the transformer network module, and the feature sequence is processed globally through a self-attention mechanism to obtain a context-enhanced feature sequence. The context-enhanced feature sequence is input into a bidirectional long short-term memory network, and the context-enhanced feature sequence is processed in the temporal dimension to obtain character prediction features corresponding to each temporal position. The character prediction features are input into the temporal classification layer to calculate the character probability distribution corresponding to each temporal position. Sequence decoding is performed based on the character probability distribution to obtain the text in the seal area.

[0088] It should be noted that the attention-guided multi-stage feature fusion stamp recognition and text extraction device provided in the second aspect of the present invention can realize all the processes of the attention-guided multi-stage feature fusion stamp recognition and text extraction method described in the first aspect above. The functions and technical effects of each module and unit in the device are the same as those of the attention-guided multi-stage feature fusion stamp recognition and text extraction method described in the first aspect above, and will not be repeated here.

[0089] A third aspect of the present invention also provides a computer-readable storage medium comprising a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the attention-guided multi-stage feature fusion stamp recognition and text extraction method described in any of the first aspects of the present invention.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary hardware platforms, and of course, it can also be implemented entirely by hardware. Based on this understanding, all or part of the technical solution of the present invention that contributes to the background technology can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0091] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for seal recognition and text extraction based on attention-guided multi-stage feature fusion, characterized in that, include: An image containing a seal is input into an improved YOLOv10 model to extract the seal region from the image. The improved YOLOv10 model is a new model formed by adding an attention guidance module after the feature extraction module of the YOLOv10 backbone network. Input the seal area into the seal text extraction model to extract the text in the seal area.

2. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 1, characterized in that, The stamp area in the image is extracted using the following steps: The first feature matrix output by the feature extraction module is compressed to obtain several attention matrices, and the several attention matrices are fused to obtain the first attention weight matrix; Perform matrix multiplication between the first attention weight matrix and the first feature matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed, and a second attention weight vector is generated through a weight generation unit; The second attention weight vector is activated and then multiplied with the first enhanced feature matrix to obtain the attention enhanced feature matrix. The stamp region in the image is extracted based on the attention-enhanced feature matrix.

3. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 2, characterized in that, The first feature matrix output by the feature extraction module is compressed to obtain several attention matrices, and the several attention matrices are fused to obtain a first attention weight matrix, including: The first feature matrix output by the feature extraction module is compressed using the global average pooling layer and the global max pooling layer in the attention guidance module, respectively, to obtain the first attention matrix and the second attention matrix. The first attention matrix and the second attention matrix are concatenated, and a convolution operation is performed on the concatenated matrix to obtain an intermediate feature matrix; The intermediate feature matrix is ​​subjected to Softmax activation to obtain the first attention weight matrix.

4. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 2, characterized in that, The step of compressing the first enhanced feature matrix and generating a second attention weight vector through a weight generation unit includes: Perform matrix multiplication on the first feature matrix and the first attention weight matrix to obtain the first enhanced feature matrix; The first enhanced feature matrix is ​​compressed using a global max pooling layer to obtain the third attention matrix; The third attention matrix is ​​processed using a fully connected layer to obtain the second attention weight vector.

5. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 1, characterized in that, The step of inputting the seal area into the seal text extraction model and extracting the text in the seal area includes: The stamp region is input into a feature multiplexing encoder to extract the enhanced fusion features of the stamp region; The enhanced fusion features are input into the feature decoder to extract the text in the stamp area.

6. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 5, characterized in that, The feature reuse encoder is a multi-stage feature extraction network built on ResNet-50. The enhanced fusion features of the stamp region are extracted using the feature reuse encoder through the following steps: The feature multiplexing encoder is used to extract features from the stamp area to obtain first image features, and convolution operation is performed on the first image features to obtain the first image features after dimensionality reduction. Feature extraction is performed on the first image features to obtain the second image features, and convolution operation is performed on the second image features to obtain the dimension-reduced second image features; The second image features are extracted to obtain the third image features, and the third image features are convolved to obtain the dimension-reduced third image features. The third image features are extracted to obtain the fourth image features; The first image feature, the second image feature, the third image feature, and the fourth image feature after dimensionality reduction are concatenated, and the concatenated feature data is convolved to obtain the enhanced fusion feature.

7. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 5, characterized in that, The step of inputting the enhanced fusion features into the feature decoder to extract the text in the stamp region includes: The enhanced fusion features are serialized to obtain a feature sequence; The feature sequence is input into the transformer network module, and the feature sequence is processed globally through a self-attention mechanism to obtain a context-enhanced feature sequence. The context-enhanced feature sequence is input into a bidirectional long short-term memory network, and the context-enhanced feature sequence is processed in the temporal dimension to obtain character prediction features corresponding to each temporal position. The character prediction features are input into the temporal classification layer to calculate the character probability distribution corresponding to each temporal position. Sequence decoding is performed based on the character probability distribution to obtain the text in the seal area.

8. The method for seal recognition and text extraction based on attention-guided multi-stage feature fusion as described in claim 7, characterized in that, The connection-time classification layer uses the negative log-likelihood of conditional probability as the loss function and optimizes the parameters of the transformer network module and the bidirectional long short-term memory network through backpropagation.

9. A multi-stage feature fusion stamp recognition and text extraction device based on attention guidance, used to implement the multi-stage feature fusion stamp recognition and text extraction method based on attention guidance as described in any one of claims 1 to 8, the device comprising: The stamp region extraction module is used to input an image containing a stamp into an improved YOLOv10 model and extract the stamp region in the image. The improved YOLOv10 model is a new model formed by adding an attention guidance module after the feature extraction module of the YOLOv10 backbone network. The seal text extraction module is used to input the seal area into the seal text extraction model and extract the text in the seal area.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform a multi-stage feature fusion stamp recognition and text extraction method based on attention guidance as described in any one of claims 1 to 8.

Images