Multi-category card image collaborative segmentation method and device

Abstract

The invention provides a multi-category card image collaborative segmentation method and device, and relates to the technical field of image processing, and the method comprises the steps: inputting a plurality of card images of the same category, and extracting a feature map of each image through an encoder; dividing the feature map into a plurality of image blocks, and calculating a collaborative attention weight between the image blocks through a Transform model; performing element-level weighted fusion on the feature map based on the collaborative attention weight, generating a weighted feature map, and inputting the weighted feature map into a ConvLSTM model for time sequence feature screening; and inputting the screened feature map into a decoder to generate a multi-category card segmentation result containing the public semantic region. According to the collaborative segmentation method for the multi-category card images, the segmentation accuracy of the multi-category card images is remarkably improved, the problem that labels are wrongly labeled due to similar appearances of the cards is effectively solved, and the error rate of subsequent OCR recognition and business processing is reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for collaborative segmentation of images of multiple types of cards. Background Technology

[0002] Image processing technology, as a crucial support for the automation of modern financial operations, is widely used in card recognition scenarios in insurance, banking, and other fields. With the development of deep learning technology, collaborative segmentation methods based on Siamese networks have gradually become the mainstream technical solution. This approach constructs a technical system for multi-image semantic feature processing through the collaborative operation of feature extraction, comparative analysis, and region partitioning. Specifically, this technology covers the entire process from image input to feature encoding, attention calculation, and feature fusion, including key components such as the ResNet50 encoder, feature map comparison module, and label propagation mechanism. Traditional methods extract salient features from multiple card images for comparison and use feature similarity calculation to achieve preliminary segmentation of common regions, but their technical architecture still has significant limitations.

[0003] However, existing collaborative segmentation methods for card images directly employ a combination of feature comparison and label propagation without establishing an effective category differentiation mechanism. This can lead to technical defects such as feature confusion and attention defocusing. Specifically, in the feature extraction stage, the ResNet50 encoder struggles to effectively separate the semantic features of similar cards (such as credit cards from different banks), causing mutual interference of category information in the feature maps. In the attention calculation stage, the attention weights of traditional collaborative segmentation networks cannot accurately capture common semantic regions across images, resulting in information loss during feature fusion. Furthermore, existing technologies lack modeling of the temporal relationships between multiple images, making it impossible to fully eliminate noise interference during feature selection. These technical problems directly lead to mislabeling in the segmentation results, with measured mislabeling rates as high as 3-5%, which in turn triggers a chain reaction of problems such as OCR recognition errors and business process interruptions, significantly impacting the efficiency of automated processing in financial services and customer experience. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a collaborative segmentation method for multi-category card images.

[0006] The second objective of this invention is to provide a collaborative segmentation device for images of multiple card types.

[0007] The third objective of this invention is to provide an electronic device.

[0008] The fourth objective of this invention is to provide a computer-readable storage medium.

[0009] The fifth objective of this invention is to provide a computer program product.

[0010] To achieve the above objectives, a first aspect of the present invention proposes a collaborative segmentation method for multi-category card images, comprising: S1, Input multiple images of cards of the same category, and use an encoder to extract the feature map of each image; S2 divides the feature map into multiple image blocks and calculates the collaborative attention weights between the image blocks using the Transformer model; S3, based on the collaborative attention weights, the feature maps are fused element-wise to generate a weighted feature map, which is then input into the ConvLSTM model for temporal feature filtering; S4. Input the filtered feature map into the decoder to generate multi-class card segmentation results containing common semantic regions.

[0011] Optionally, in S11, a ResNet50 encoder is used to extract features at multiple scales for each card image, generating feature maps containing semantic information at different levels. S12 uses a feature pyramid structure to perform cross-layer fusion of the feature maps output by ResNet50, thereby enhancing the semantic expressive power of the feature maps.

[0012] Optionally, S21, the feature map is divided into segments of size [size missing]. Non-overlapping image patches; S22, attention weights across image patches are calculated using the Transformer's QKV mechanism, where the query vector... Key vector Sum value vector The dimensions are respectively , and It employs a multi-head attention mechanism, with a head count of [number missing]. .

[0013] Optional, S31, via the forget gate of ConvLSTM and input gate The weighted feature maps are filtered, among which , ; S32, the filtered feature map is passed through the output gate of ConvLSTM. Generate the final temporal feature representation, where .

[0014] Optionally, in S41, a transposed convolutional layer is used to upsample the filtered feature map to restore it to the original image resolution; S42 uses a multi-layer decoder structure to perform layer-by-layer feature fusion on the upsampled feature map, and finally outputs pixel-level segmentation results for multiple types of cards.

[0015] Optionally, S5 performs post-processing on the segmentation results, removing isolated noise points in the segmented region through morphological operations, and smoothing the segmentation contour using a boundary optimization algorithm. The morphological operations include opening and closing operations, and the boundary optimization algorithm adopts a contour fitting method based on minimum spanning tree.

[0016] To achieve the above objectives, a second aspect of the present invention provides a multi-category card image collaborative segmentation device, comprising: The image input and feature extraction module is used to input multiple images of cards of the same category and extract the feature map of each image using an encoder; The image patch segmentation and collaborative attention calculation module is used to divide the feature map into multiple image patches and calculate the collaborative attention weights between each image patch using the Transformer model. The element-level weighted fusion and ConvLSTM temporal feature filtering module is used to perform element-level weighted fusion of feature maps based on the collaborative attention weights, generate weighted feature maps, and then input them into the ConvLSTM model for temporal feature filtering, wherein the forgetting gate of ConvLSTM is used. and input gate Filter the weighted feature maps. , ; The decoder segmentation module is used to input the filtered feature maps into the decoder to generate multi-class card segmentation results containing common semantic regions.

[0017] To achieve the above objectives, a third aspect of the present invention provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of the first aspects.

[0018] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of the first aspects.

[0019] To achieve the above objectives, a fifth aspect of the present invention provides a computer program product that, when executed by a processor, implements the method described in any one of the first aspects.

[0020] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: The methods, apparatus, electronic devices, and computer-readable storage media of the present invention significantly improve the segmentation accuracy of multi-category card images, effectively solve the problem of label mislabeling caused by similar card appearances, and reduce the error rate of subsequent OCR recognition and business processing.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating a multi-category card image collaborative segmentation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall operation process provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the structure of a multi-category card image collaborative segmentation device provided in an embodiment of the present invention. Detailed Implementation

[0023] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0024] To address the issue of inaccurate segmentation labels caused by similar card types, this invention provides a collaborative segmentation method for multi-category card images. Figure 1 This is a flowchart illustrating a multi-category card image collaborative segmentation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall operation process provided in an embodiment of the present invention. Figure 1 , Figure 2 As shown, the method includes the following steps: S1: Input multiple images of cards of the same category and use an encoder to extract the feature map of each image.

[0025] Specifically, in some implementations, inputting multiple card images of the same category and extracting feature maps from each image using an encoder is one of the key steps in the collaborative segmentation network of this proposal for processing multi-category card images. This step aims to provide high-quality input features for subsequent attention computation and semantic region selection through a unified feature extraction mechanism.

[0026] In terms of technical implementation, this step uses ResNet-50 as the encoder. Its structure is based on a deep convolutional neural network with a 50-layer stacked structure, effectively extracting multi-scale semantic features from images. After normalization, the input image is fed into the ResNet-50 network, where multiple convolutional and pooling layers progressively extract feature maps from low-level edge features to high-level semantic features. The feature map output for each image is... The three-dimensional tensor, in which and These represent the height and width of the feature map, respectively. This indicates the number of channels, typically 2048.

[0027] In terms of parameter indicators, the resolution of the input image is uniformly adjusted to This ensures the alignment of feature maps across different images. The output size of ResNet-50 feature maps is typically [size missing]. That is, when the input image size is At this point, after five downsampling layers, the spatial resolution of the feature map is compressed to 1 / 32 of the original image. Furthermore, to enhance model robustness, the input image undergoes image enhancement processing before entering the encoder, including random flipping, Gaussian blurring, and sharpening, thereby increasing data diversity.

[0028] At the application level, this step is suitable for image processing scenarios involving multiple cards (such as ID cards, bank cards, driver's licenses, etc.) in insurance business. In actual deployment, the system will input a group of card images of the same category in batches (e.g., 3 to 5 images), and extract their shared semantic features through the encoder to provide input for the subsequent Transformer attention mechanism.

[0029] In terms of technical effectiveness, the feature maps extracted by the ResNet-50 encoder in this step have strong semantic expressive power and can effectively capture key regions (such as text, borders, logos, etc.) in card images, providing a solid foundation for subsequent collaborative attention calculation and ConvLSTM semantic filtering, thereby improving the collaborative segmentation accuracy of multi-class card images.

[0030] Furthermore, S1 includes: S11 uses a ResNet50 encoder to extract features at multiple scales for each card image, generating feature maps containing semantic information at different levels.

[0031] Specifically, in some implementations, the step of "using a ResNet50 encoder to perform multi-scale feature extraction on each card image and generating feature maps containing semantic information at different levels" in this application proposal is a key step in feature extraction within the entire collaborative segmentation network. ResNet50, as a classic convolutional neural network architecture, primarily addresses the vanishing gradient problem in deep networks through residual connections, thereby improving the model's training stability and feature representation capabilities. This encoder consists of multiple convolutional layers, batch normalization layers, and ReLU activation functions, containing a total of 50 convolutional layers. The first four layers (Conv1-Conv4) correspond to feature extraction at different scales, and the output feature maps possess multi-level information ranging from low-level edges and textures to high-level semantics.

[0032] In the specific implementation, each card image is first input into the ResNet50 backbone network. After layer-by-layer convolution operations, multi-level feature maps are output. For example, the feature map output by layer 4 (Conv4) is typically 1 / 16 the size of the input image with 1024 channels, while the feature map output by layer 5 (Conv5) is 1 / 32 the size with 2048 channels. These feature maps capture the semantic information of the image at different scales, providing multi-granular feature representations for subsequent collaborative attention computation.

[0033] In application scenarios, this step is mainly used to process multi-category card images (such as ID cards, bank cards, insurance policy cards, etc.) uploaded by users. Under complex background or lighting conditions, ResNet50's multi-scale feature extraction capability can effectively enhance the model's perception of key card regions, thereby improving the robustness and accuracy of segmentation.

[0034] Furthermore, the technical effect of this step is that, through multi-scale feature extraction of ResNet50, it can provide the Transformer module with feature patches with rich semantics, thereby enhancing the model's ability to recognize common semantic regions in multiple images and significantly improving the collaborative segmentation accuracy of multi-class cards.

[0035] S12 uses a feature pyramid structure to perform cross-layer fusion of the feature maps output by ResNet50, thereby enhancing the semantic expressive power of the feature maps.

[0036] Specifically, in some implementations, feature maps output by ResNet50 are fused across layers using a feature pyramid structure. This aims to enhance the semantic expressiveness of the feature maps, thereby improving the collaborative segmentation accuracy of multi-class card images. This step is technically based on a multi-scale feature fusion mechanism, utilizing feature maps output by the ResNet50 network at different layers, each corresponding to image regions with different receptive fields and semantic richness. Specifically, ResNet50 typically outputs feature maps at layers such as conv2, conv3, conv4, and conv5. conv5 possesses the strongest semantic information but has lower spatial resolution, while conv2 has higher spatial resolution but weaker semantics. Through a feature pyramid structure, these feature maps from different layers are upsampled, concatenated, or weighted and fused to construct a fused feature map with multi-scale semantic information.

[0037] At the parameter level, the feature map sizes output by ResNet50 are typically H / 32 × W / 32 (conv5), H / 16 × W / 16 (conv4), H / 8 × W / 8 (conv3), and H / 4 × W / 4 (conv2), where H and W are the height and width of the input image, respectively. During cross-layer fusion, bilinear interpolation or transposed convolution is typically used for upsampling to align the feature map sizes across layers. Fusion methods may include concatenation or weighted summation, where weighted summation can incorporate learnable weight parameters. This is used to dynamically adjust the contribution ratio of features at each level, such as: ,in Indicates the first Feature maps of the hierarchy.

[0038] At the application level, this step is mainly used in collaborative segmentation tasks involving multiple card images. By fusing multi-level features, the model can more accurately identify common semantic regions of the cards, especially when card edges are blurred, backgrounds are complex, or occlusions exist. Cross-level fusion helps preserve detailed information and enhance semantic consistency. This step is typically deployed before the Transformer model as a key step in feature extraction and enhancement.

[0039] In terms of technical effectiveness, this step significantly improves the multi-scale expressive power of feature maps, enabling the subsequent Transformer model to more comprehensively capture the common area information of cards when calculating collaborative attention, thereby reducing mislabeling and category confusion problems and improving the robustness and accuracy of the segmentation model.

[0040] S2 divides the feature map into multiple image blocks and calculates the collaborative attention weights between the image blocks using the Transformer model.

[0041] Specifically, in some implementations, dividing the feature map into multiple image patches and calculating the collaborative attention weights between these patches using a Transformer model is a key step in improving the collaborative segmentation accuracy of multi-class card images in this invention. This step is technically implemented by using shared semantic feature maps extracted from multiple card images via convolutional neural networks (such as ResNet-50), and converting them into multiple local image patches through a spatial partitioning strategy, thereby providing structured input for the subsequent modeling of the attention mechanism.

[0042] Specifically, feature maps are typically partitioned using a sliding window or non-overlapping blocks, dividing the input feature map... According to the preset block size (e.g.) or Divided into several local regions ,in and These represent the height and width of the image block, respectively. This represents the number of channels in the feature map. In this invention, a non-overlapping block method may be optionally used to reduce redundant calculations while ensuring the integrity of the features.

[0043] Then, all image blocks As the input sequence to the Transformer model, the collaborative attention weights between blocks are calculated using a self-attention mechanism. The formula for calculating the attention weights in the Transformer model is:

[0044] in, , , These are query, key, and value matrices, respectively. denoted as the dimension of the key vector. Through this mechanism, the model can dynamically learn the semantic relevance between different image patches, thereby enhancing its focus on common semantic regions.

[0045] In terms of parameter settings, the Transformer module typically includes a multi-head attention mechanism, with 8 or 12 heads, an embedding dimension of 256, and a feedforward network hidden layer dimension of 512. This step is particularly suitable for scenarios where multiple card images contain similar structures but different categories. For example, different types of insurance cards may appear highly similar, but their key information regions differ. Guided by attention weights, the model can more accurately distinguish these regions, thereby improving the consistency of segmentation categories and boundary accuracy.

[0046] The technical effect of this step is that by introducing the Transformer model to model the semantic relationships between image patches, it effectively alleviates the missegmentation problem caused by the similarity of card categories in traditional collaborative segmentation methods, and significantly improves the accuracy and robustness of the segmentation results.

[0047] Furthermore, S2 includes: S21, divide the feature map into sections of size [size missing]. Non-overlapping image patches.

[0048] Specifically, in some implementations, the feature map is divided into sections of size [size missing]. Non-overlapping image patches are a key step in this proposal for extracting local semantic information and computing collaborative attention. This step is typically performed after the feature map has been processed by an encoder (such as ResNet50) to extract high-level semantic features. Its purpose is to divide the global feature map into multiple semantically consistent local regions (patches) so that the subsequent Transformer model can more effectively model cross-image attention relationships between multiple images.

[0049] Specifically, this partitioning operation performs a uniform slice based on the spatial dimension of the feature map. Assume the feature map size output after ResNet50 encoding of the input image is... ,in and The height and width of the feature map, For the number of channels. In this proposal, the feature map is arranged according to... The window size is used to divide the image into non-overlapping sections, meaning there is no pixel overlap between each image patch, with a window stride of 16. This allows the feature map to be divided into... There are 1 patch, and the size of each patch is 1. This partitioning method ensures that each patch has relatively independent semantic information in space, while maintaining the structural integrity of the feature map.

[0050] In practical applications, this step is typically implemented on the GPU using tensor operations, such as using the `unfold` or `view` operations in PyTorch for segmentation. Since the Transformer model is sensitive to the length of the input sequence, the number and size of patches need to be preset during the model design phase to ensure the efficiency and stability of subsequent attention calculations.

[0051] The technical advantage of this step is that by dividing the feature map into fixed-size patches, local regions of multiple images can be aligned, thus providing the Transformer with a structured input sequence, facilitating its computation of cross-image collaborative attention. Furthermore, this partitioning method helps the model focus on regions with common semantics, improving the accuracy and robustness of collaborative segmentation of multi-class card images.

[0052] S22, attention weights across image patches are calculated using the Transformer's QKV mechanism, where the query vector... Key vector Sum value vector The dimensions are respectively , and It employs a multi-head attention mechanism, with a head count of [number missing]. .

[0053] Specifically, in some implementations, this proposal utilizes the Transformer's QKV (Query-Key-Value) mechanism to compute attention weights across image patches, thereby enhancing the model's ability to focus on common semantic regions across multiple card images. Specifically, this step first extracts feature maps from the input image using a ResNet50 encoder, then divides the feature maps into several image patches, each patch being mapped to a feature vector. Within the Transformer module, the feature vector of each patch is used to generate query vectors through a learnable linear transformation layer. Key vector Sum value vector Their dimensions are respectively , and This design ensures semantic alignment between queries and keys while preserving the high-dimensional expressiveness of value vectors to support richer feature reconstruction.

[0054] Furthermore, this scheme employs a multi-head attention mechanism, with the number of heads set to [value missing]. Each attention head is calculated independently. , , The attention weights between them are calculated using the scaled dot product attention formula. ,in The dimension of the key vector (i.e. This mitigates the problem of dot product results amplifying with increasing dimensionality. The outputs of all heads are concatenated and then linearly transformed again to obtain the final attention output feature map.

[0055] In application scenarios, this step is mainly used for collaborative segmentation tasks of multi-category card images, such as the joint processing of driver's license, vehicle registration, and bank card images in insurance business. Through attention calculation across image patches, the model can identify and focus on semantically consistent regions in multiple images, thereby improving the accuracy and robustness of segmentation.

[0056] The technical effect of this step is that by introducing the attention mechanism of Transformer, the model can effectively capture the semantic relationship between multiple images, reduce the missegmentation problem caused by the similarity of card appearance, and significantly improve the category accuracy and boundary clarity of the segmentation results.

[0057] S3, based on the collaborative attention weights, the feature maps are fused element-wise, and the weighted feature maps are then input into the ConvLSTM model for temporal feature filtering.

[0058] Specifically, in some implementations, element-wise weighted fusion of feature maps based on the aforementioned collaborative attention weights is used to generate a weighted feature map, which is then input into a ConvLSTM model for temporal feature filtering. This is one of the key steps in achieving collaborative segmentation of multi-category card images in this proposal. The core of this step lies in using the collaborative attention weights calculated by the Transformer model to perform weighted fusion of feature maps from multiple images, thereby enhancing the model's ability to focus on common semantic regions. Then, the ConvLSTM model is used to perform temporal modeling on the fused feature map to further extract common semantic information with temporal consistency.

[0059] Specifically, after the ResNet50 encoder extracts the feature map for each image, the feature map is divided into several non-overlapping image patches, which are then fed into the Transformer model. The Transformer calculates the co-attention weights between patches using self-attention and cross-attention mechanisms. These weights reflect the correlation of the same semantic regions in different images. In some implementations, the calculation of the attention weights can be expressed as:

[0060] in, and The first The query vector of the first image patch and the first The key vector of each image patch, This represents the dimension of the key vector. Through this mechanism, the model can automatically focus on semantically consistent regions across multiple images, such as card edges or text regions.

[0061] In terms of parameter settings, the Transformer module typically employs a multi-head attention mechanism, with 8 or 16 heads, an embedding dimension of 256, and 2 to 4 layers. The weighted fusion process uses element-wise multiplication to fuse the attention weight map with the original feature map, generating a weighted feature map with a dimension of [missing information]. ,in , The height and width of the feature map, This represents the number of channels.

[0062] Subsequently, the weighted feature map is input into a ConvLSTM model, which uses a gating mechanism to perform temporal modeling of the feature map and extract common semantic regions with temporal consistency. The number of input channels of ConvLSTM is related to the weighted feature map. Figure 1 The number of channels for both hidden and cell states can be set to 128-256, with 2-4 time steps. Through ConvLSTM processing, the model can further suppress noise and inconsistent regions, enhancing the temporal stability of the target region.

[0063] In practical applications, this step is particularly suitable for situations where multiple card images contain similar structures or background interference, such as different types of bank cards, ID cards, and driver's licenses. By combining attention mechanisms and temporal modeling, the model can more accurately identify and segment common target regions, thereby significantly improving segmentation accuracy and robustness.

[0064] Furthermore, S3 includes: S31, through the forget gate of ConvLSTM and input gate The weighted feature maps are filtered, among which , .

[0065] Specifically, in some implementations, this proposal utilizes the forget gate of ConvLSTM. and input gate The weighted feature maps output by the Transformer module are further filtered to enhance the model's ability to focus on common semantic regions in multiple card images. Specifically, ConvLSTM introduces a time-series modeling mechanism to dynamically fuse and optimize the semantic features of multiple images, thereby improving the accuracy and robustness of collaborative segmentation.

[0066] From a technical implementation perspective, the forget gate of ConvLSTM and input gate Used to control historical states and cell state For the current input The degree of retention and updating. Its mathematical expression is:

[0067]

[0068] in, This represents the Sigmoid activation function, used to compress the gate values ​​to the [0,1] interval. and These are the learnable weight matrices for the forget gate and the input gate, respectively. This indicates that the hidden state from the previous moment will be restored. Cell state Weighted feature map at the current time A concatenation operation is performed to form a 3D input tensor. This operation concatenates along the channel dimension, typically implemented using a `Concatenate` layer. The resulting tensor has a size of [size missing]. ,in , The height and width of the feature map, , , They are respectively , and The number of channels.

[0069] In terms of parameter metrics, weight matrix and Typically, a 3×3 convolutional kernel is used for parameter learning, with a stride of 1 and `same` padding to maintain the spatial resolution of the feature map. The output of the Sigmoid function determines which feature information needs to be retained or updated, thus achieving dynamic filtering of the weighted feature map. In actual training, this step helps suppress noisy regions and irrelevant semantics, enhancing the model's ability to perceive common areas of the cards.

[0070] In terms of application scenarios, this step is mainly used in collaborative segmentation tasks for multi-category card images, especially when the input consists of multiple similar but different card images. Through the gating mechanism of ConvLSTM, the model can more accurately distinguish the semantic boundaries of different cards, reducing mislabeling and misclassification. For example, in insurance claims scenarios, different types of bank cards, ID cards, and insurance policy cards may have similar appearances, and this step can effectively improve the model's ability to identify key regions.

[0071] In terms of technical effectiveness, this step enhances the model's ability to model common semantic regions in multiple images by introducing the gating mechanism of ConvLSTM, thereby significantly improving the accuracy and consistency of the segmentation results and solving the problem of misclassification of labels caused by card similarity in existing technologies.

[0072] S32, the filtered feature map is passed through the output gate of ConvLSTM. Generate the final temporal feature representation, where .

[0073] Specifically, in some implementations, the filtered feature maps are passed through the output gate of a ConvLSTM. Generating the final temporal feature representation is one of the key steps in this proposal to enhance the collaborative segmentation performance of multi-class card images. This step is based on the ConvLSTM (Convolutional Long Short-Term Memory) structure. By introducing a temporal modeling mechanism, it further semantically filters and fuses the weighted feature maps output by the Transformer module, thereby extracting more discriminative common semantic region features.

[0074] Specifically, ConvLSTM dynamically processes the input features through its gating mechanism, where the output gate... Used to control the hidden state at the current moment. Impact on output. The formula for calculating the output gate is:

[0075] in, This represents the Sigmoid activation function. Let be the convolution weight matrix of the output gate. and These represent the hidden state and the cell state from the previous moment, respectively. This is the input feature map at the current time step. The formula achieves dynamic modeling of temporal information by concatenating the three feature maps along the channel dimension and then extracting the gating signal through convolution operations.

[0076] In practical applications, this step is typically deployed after the Transformer module to process the sequence of feature maps weighted by an attention mechanism from multiple card images. The size of the input feature maps is usually [size missing]. ,in and The height and width of the feature map, This represents the number of channels. The kernel size for ConvLSTM is typically set to [value missing]. The step size is 1, and padding is used to keep the feature map size constant. The output of the output gate... For a hidden state Binarized gating maps of the same size are used to control the activation level of the final output feature map.

[0077] The technical advantage of this step lies in its ability to effectively capture the semantic consistency of multiple card images across spatial and temporal dimensions through the temporal modeling capabilities of ConvLSTM. This further suppresses the influence of noise and irrelevant regions, thereby improving the segmentation model's accuracy and robustness in recognizing multiple card categories. In this proposal, this step works in conjunction with the Transformer module to achieve accurate localization and segmentation of common semantic regions in multiple images.

[0078] S4. Input the filtered feature map into the decoder to generate multi-class card segmentation results containing common semantic regions.

[0079] Specifically, in some implementations, inputting the filtered feature maps into the decoder to generate multi-class card segmentation results containing common semantic regions is one of the key steps in achieving collaborative image segmentation in this proposal. The core of this step lies in upsampling and semantically mapping the feature maps after semantic filtering by the Transformer attention mechanism and ConvLSTM through the decoder structure, thereby outputting an accurate segmentation mask map, which is used to identify and distinguish card regions of the same category in multiple images.

[0080] In terms of technical implementation, this step typically uses U-Net or its variants as the decoder structure. The decoder progressively recovers the spatial resolution of the feature maps through deconvolution or transposed convolution operations, while fusing skip connection information from the encoder to preserve more detailed features. In each layer of the decoder, the feature maps undergo nonlinear transformations through multiple convolutional layers, ultimately outputting a segmentation result with the same size as the input image. In this proposal, the input to the decoder is a weighted feature map output from a ConvLSTM, typically with 256 or 512 channels and a spatial size of 1 / 16 or 1 / 32 of the input image, depending on the encoder's downsampling factor.

[0081] In terms of parameter metrics, the number of output channels of the decoder is consistent with the number of card categories. For example, if the system needs to identify 5 card categories, then the number of output channels is 5. Pixel-level classification of the segmentation results typically uses the Softmax activation function to ensure that the category probability distribution of each pixel satisfies the normalization condition. The accuracy of the segmentation mask can be evaluated using metrics such as Intersection over Union (IoU) or the Dice coefficient, where the Dice coefficient is defined as:

[0082] in To predict segmentation regions, This represents the actual labeled area.

[0083] In terms of application scenarios, this step is widely used in multi-category card recognition scenarios in insurance business, such as driver's licenses, vehicle registration certificates, ID cards, and bank cards. In actual deployment, the segmentation results output by the decoder can be directly used for cropping OCR recognition regions, anomaly detection in image quality inspection, and subsequent structured data extraction processes.

[0084] In terms of technical effectiveness, this step uses a decoder to restore semantic information into pixel-level segmentation results, which significantly improves the segmentation accuracy and category consistency of multi-category cards under multi-image input, effectively reduces subsequent processing deviations caused by mislabeling, and thus improves the robustness and business processing efficiency of the overall system.

[0085] The multi-category card image collaborative segmentation method of this invention significantly improves the segmentation accuracy of multi-category card images and effectively solves the problem of mislabeling caused by similar card types.

[0086] Furthermore, S4 includes: S41 uses a transposed convolutional layer to upsample the filtered feature maps, restoring them to the original image resolution.

[0087] Specifically, in this application, the use of transposed convolutional layers to upsample the filtered feature maps is one of the key steps in achieving collaborative segmentation of multi-class card images. This step aims to restore the high-level semantic feature maps filtered by the Transformer attention mechanism and ConvLSTM to the resolution of the original image, thereby providing accurate feature mapping for the final pixel-level segmentation.

[0088] In some implementations, this upsampling process is accomplished through a transposed convolution operation, which is essentially the inverse of a convolution operation, used to expand the spatial dimension of the feature map. Specifically, the size of the input feature map is... The goal is to restore it to the original image size. Transposed convolutional layers are typically configured with a stride. kernel size ,filling The structure is designed to achieve a 2x upsampling ratio. In practical deployments, multiple transposed convolutional layers can be optionally used for progressive upsampling, with batch normalization and ReLU activation functions added between each layer to enhance the non-linearity of feature representation and stabilize the training process.

[0089] In terms of parameter metrics, the number of output channels of the transposed convolutional layer should be consistent with the number of channels of the previous layer in the decoder to ensure the continuity of the feature map in the channel dimension. For example, if the feature map output by ConvLSTM has 64 channels, then the output channels of the transposed convolutional layer should also be set to 64. Furthermore, the resolution recovery accuracy of the feature map needs to be controlled during upsampling. Typically, the spatial error between the final output feature map and the original input image should be less than 5%, which satisfies the requirement... This is to ensure the geometric consistency of the segmentation results.

[0090] At the application level, this step is mainly used in the decoder module to gradually restore the semantically filtered feature map to the same resolution as the input image, so as to perform pixel-level segmentation prediction. In card image processing, since cards usually have fixed structures and clear edges, high-precision upsampling helps to preserve key edge information, thereby improving the quality of segmentation boundaries and the accuracy of category recognition.

[0091] In terms of technical effectiveness, this step effectively alleviates the information loss problem caused by the loss of feature map resolution through the structured design of transposed convolutional layers, enabling the model to more accurately locate the common semantic regions of cards and significantly improve the segmentation accuracy and robustness of multi-class card images.

[0092] S42 uses a multi-layer decoder structure to perform layer-by-layer feature fusion on the upsampled feature map, and finally outputs pixel-level segmentation results for multiple types of cards.

[0093] Specifically, in some implementations, a multi-layer decoder structure is used to perform layer-by-layer feature fusion on the upsampled feature map, ultimately outputting pixel-level segmentation results for multiple card categories. This is a key step in achieving high-precision collaborative image segmentation in this proposal. This step is based on the feature map jointly processed by the Transformer model and the ConvLSTM model. The decoder structure gradually restores the spatial resolution of the image and fuses semantic information at different scales to improve the accuracy and boundary integrity of the segmentation results.

[0094] At the technical implementation level, decoders typically employ structures such as U-Net or FPN (Feature Pyramid Network), consisting of multiple upsampling modules and skip connections. Each upsampling module progressively restores the feature map from low resolution to the original image size through transposed convolution or interpolation operations. During layer-by-layer fusion, the upsampled feature map of the current layer is concatenated or element-wise added with the feature map of the corresponding layer in the encoder, thereby preserving more detailed information. Furthermore, the fused feature map undergoes non-linear transformation and feature extraction through convolutional layers, enhancing the model's ability to perceive card edges and textures.

[0095] At the parameter level, the number of channels in the convolutional layers of the decoder is usually consistent with or gradually decreases from the corresponding layers in the encoder, for example, gradually decreasing from 256 channels to 64 channels, ultimately outputting the same number of channels as the number of categories (e.g., 8 channels for 8 card categories). The upsampling factor is generally a power of 2, such as 2×, 4×, or 8×, to ensure gradual restoration of spatial resolution. During training, the segmentation map output by the decoder is compared pixel-by-pixel with the ground truth label map, and optimization is performed using cross-entropy loss or Dice loss to improve the category accuracy and boundary consistency of the segmentation.

[0096] At the application level, this step is widely used in multi-category card recognition scenarios in insurance business, such as driver's licenses, vehicle registration certificates, ID cards, and bank cards. In actual deployment, the input is a set of multi-category card images, and the decoder output is a pixel-level segmentation mask for each image, used for subsequent OCR recognition or image quality inspection tasks. Through layer-by-layer fusion, the model can more accurately distinguish card types, reduce mislabeling and omissions, thereby improving the automation level and processing efficiency of the overall business process.

[0097] The technical effect of this step is that, through the multi-layer decoder structure and feature fusion mechanism, it significantly enhances the model's ability to recognize common semantic regions in card images, improves the accuracy and robustness of the segmentation results, solves the problem of misclassification of labels caused by similar card categories in existing technologies, and provides a high-quality input foundation for subsequent image processing.

[0098] The multi-category card image collaborative segmentation method of this invention significantly improves the segmentation accuracy of multi-category card images and effectively solves the problem of mislabeling caused by similar card types.

[0099] S5 performs post-processing on the segmentation results, removing isolated noise points in the segmented region through morphological operations, and smoothing the segmentation contour using a boundary optimization algorithm. The morphological operations include opening and closing operations, and the boundary optimization algorithm adopts a contour fitting method based on minimum spanning tree.

[0100] Specifically, in image segmentation tasks, the segmentation results often contain isolated noise points and contour discontinuities, which directly affect the accuracy of subsequent OCR recognition and image quality inspection. To address this issue, this proposal introduces a post-processing module after the segmentation results are output, which includes morphological operations and boundary optimization algorithms.

[0101] In some implementations, morphological operations combine opening and closing operations to remove isolated noise points in segmented regions and fill small holes in the contours. Opening, through erosion followed by dilation, effectively eliminates small noise areas. Its structural element is typically a 3×3 or 5×5 square or circular kernel, and the number of iterations is set to 1 to 3, depending on the noise density and image resolution. Closing, through dilation followed by erosion, connects adjacent segmented regions, enhancing the connectivity of the contours. Its structural element is the same as opening, and the number of iterations is generally 1. The parameter settings for morphological operations need to be adjusted according to the noise distribution of the actual image to preserve the integrity of the target region while removing noise.

[0102] Furthermore, the boundary optimization algorithm employs a contour fitting method based on Minimum Spanning Tree (MST) to smooth the boundaries of the segmented regions. This method first samples the contour points of the segmented regions to extract a set of key boundary points. Then, a distance graph between point sets is constructed, and the minimum spanning tree is calculated using Prim's or Kruskal's algorithm to obtain the optimal contour connection path. Path reconstruction using the minimum spanning tree effectively removes contour jitter and jagged edges, improving the geometric continuity and visual consistency of the segmented regions.

[0103] This step is particularly suitable for multi-class collaborative segmentation scenarios of card images in practical applications, such as joint segmentation of different types of bank cards, ID cards, and policy cards in insurance business. By combining morphological operations with boundary optimization algorithms, the accuracy and stability of the segmentation results can be significantly improved, thereby reducing the misclassification rate in subsequent processing and improving the efficiency and reliability of the overall business process.

[0104] The multi-category card image collaborative segmentation method of this invention further improves the robustness and visual continuity of the segmentation results. It effectively eliminates the interference caused by isolated noise points through morphological operations and achieves more accurate contour fitting by combining a boundary optimization algorithm based on minimum spanning tree, thereby enhancing the segmentation quality and consistency of multi-category cards in complex backgrounds.

[0105] To achieve the above embodiments, the present invention also proposes a multi-category card image collaborative segmentation device. Figure 3 This is a schematic diagram of a multi-category card image collaborative segmentation device provided in an embodiment of the present invention. Figure 3 As shown, the device includes: The image input and feature extraction module 100 is used to input multiple card images of the same category and extract the feature map of each image using an encoder; The image patch segmentation and collaborative attention calculation module 200 is used to divide the feature map into multiple image patches and calculate the collaborative attention weights between each image patch through the Transformer model. The element-level weighted fusion and ConvLSTM temporal feature filtering module is used to perform element-level weighted fusion of the feature maps based on the collaborative attention weights, generate a weighted feature map, and then input it into the ConvLSTM model for temporal feature filtering, wherein the forgetting gate of ConvLSTM is used. and input gate Filter the weighted feature maps. , ; The decoder segmentation module 400 is used to input the filtered feature map into the decoder to generate multi-class card segmentation results containing common semantic regions.

[0106] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0107] To implement the above embodiments, the present invention also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0108] To implement the above embodiments, the present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0109] To implement the above embodiments, the present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0110] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0112] This invention is intended to provide implementation schemes for users to selectively prevent the use or access to personal information data. That is, this disclosure is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0113] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0114] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0115] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0117] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0118] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0119] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0120] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0121] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0122] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-class card image co-segmentation method, characterized in that, The method comprises the following steps: S1, inputting multiple card image images of the same category, and extracting a feature map of each image by using an encoder; S2, dividing the feature map into multiple image blocks, and calculating the collaborative attention weight between each image block by using a Transformer model; S3, element-level weighted fusion of the feature map based on the collaborative attention weight, inputting the weighted feature map into a ConvLSTM model for time sequence feature screening; S4, inputting the screened feature map into a decoder to generate a multi-category card segmentation result containing a common semantic region.

2. The method of claim 1, wherein, The inputting multiple card image images of the same category and extracting a feature map of each image by using an encoder further comprises: S11, using a ResNet50 encoder to perform multi-scale feature extraction on each card image to generate a feature map containing different levels of semantic information; S12, performing cross-level fusion on the feature map output by the ResNet50 through a feature pyramid structure to enhance the semantic expression ability of the feature map.

3. The method of claim 1, wherein, The dividing the feature map into multiple image blocks and calculating the collaborative attention weight between each image block by using a Transformer model further comprises: S21, dividing the feature map into non-overlapping image blocks of size S21, dividing the feature map into non-overlapping image blocks of size S22, the attention weight across the image blocks is calculated by the QKV mechanism of the Transformer, wherein the dimensions of the query vector , the key vector and the value vector are respectively , and , and a multi-head attention mechanism is adopted with the number of heads being .

4. The method of claim 1, wherein, The element-level weighted fusion of the feature map based on the collaborative attention weight and the inputting the weighted feature map into a ConvLSTM model for time sequence feature screening further comprises: S31, pass through the forget gate of the ConvLSTM and the input gate filter the weighted feature map, wherein , ; S32, pass the screened feature map through the output gate of the ConvLSTM generating a final temporal feature representation, wherein .

5. The method of claim 1, wherein, The inputting the screened feature map into a decoder to generate a multi-category card segmentation result containing a common semantic region further comprises: S41, using a transposed convolution layer to upsample the screened feature map to restore it to the original image resolution; S42, performing layer-by-layer feature fusion on the upsampled feature map through a multi-layer decoder structure to finally output a pixel-level segmentation result of the multi-category card.

6. The method of claim 1, wherein, Further comprising: S5, post-processing the segmentation result, removing isolated noise points in the segmentation region through morphological operations, and smoothing the segmentation contour by using a boundary optimization algorithm, wherein the morphological operations include opening operation and closing operation, and the boundary optimization algorithm uses a contour fitting method based on minimum spanning tree.

7. A multi-class card image co-segmentation apparatus, comprising: The method comprises the following steps: An image input and feature extraction module is configured to input multiple card image images of the same category, and extract a feature map of each image by using an encoder; An image block division and collaborative attention calculation module is configured to divide the feature map into multiple image blocks, and calculate the collaborative attention weight between each image block by using a Transformer model; The element-level weighted fusion and ConvLSTM time sequence feature screening module is used for element-level weighted fusion of the feature map based on the collaborative attention weight, and the weighted feature map is input into the ConvLSTM model for time sequence feature screening, wherein the forgetting gate and the input gate of the ConvLSTM are used to screen the weighted feature map, , . A decoder segmentation module is configured to input the screened feature map into a decoder to generate a multi-category card segmentation result containing a common semantic region.

8. An electronic device, comprising: The method comprises the following steps: A processor and a memory connected to the processor in communication; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1-6.

10. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images