A bill recognition method and device based on machine vision

Abstract

The present invention relates to the field of machine vision technology, and discloses a bill recognition method and device based on machine vision. The method comprises: preprocessing and randomly occluding an input bill image to obtain a preprocessed bill image; performing multi-scale feature extraction on the preprocessed bill image to obtain a multi-scale feature map set; performing progressive attention feature optimization based on the multi-scale feature map set to obtain an optimized feature map; performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map; performing multi-label collaborative learning based on the semantically aligned feature map to obtain a multi-label classification result; performing feature compression and recognition processing on the multi-label classification result to obtain a bill recognition result. The present invention can effectively process damaged and deformed bills, realize multi-scale and multi-modal feature extraction and fusion, and has cross-domain migration capabilities to adapt to bills of different types and formats.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and in particular to a bill recognition method and device based on machine vision. Background Art

[0002] Traditional bill recognition methods mainly rely on template matching and rule-based methods, which face great challenges in dealing with diverse and complex bills. In recent years, advances in machine vision and deep learning technologies have brought new opportunities for bill recognition, but there are still many problems that need to be solved.

[0003] Existing bill recognition systems perform poorly when dealing with damaged, blurred or deformed bills, and are unable to cope with various complex situations in actual scenarios. Secondly, the diversity and structural complexity of bills make feature extraction and representation learning difficult, especially in cross-domain scenarios, where the differences between different types of bills further increase the difficulty of recognition. The various types of information contained in bills (such as text, images, tables, etc.) require comprehensive analysis and understanding, but existing methods are still insufficient in multimodal information fusion and semantic understanding. Summary of the invention

[0004] The present invention provides a bill recognition method and device based on machine vision, which can effectively process damaged and deformed bills, realize multi-scale and multi-modal feature extraction and fusion, and have cross-domain migration capabilities to adapt to bills of different types and formats.

[0005] In a first aspect, the present invention provides a bill recognition method based on machine vision, the bill recognition method based on machine vision comprising:

[0006] Performing preprocessing and random occlusion processing on the input bill image to obtain a preprocessed bill image;

[0007] Performing multi-scale feature extraction on the pre-processed bill image to obtain a multi-scale feature map set;

[0008] Performing progressive attention feature optimization based on the multi-scale feature map set to obtain an optimized feature map;

[0009] Performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map;

[0010] Perform multi-label collaborative learning based on the semantic alignment feature map to obtain a multi-label classification result;

[0011] The multi-label classification result is subjected to feature compression and recognition processing to obtain a bill recognition result.

[0012] In a second aspect, the present invention provides a bill recognition device based on machine vision, the bill recognition device based on machine vision comprising:

[0013] A preprocessing module, used for preprocessing and randomly blocking the input bill image to obtain a preprocessed bill image;

[0014] An extraction module, used for performing multi-scale feature extraction on the pre-processed bill image to obtain a multi-scale feature map set;

[0015] An optimization module, configured to perform progressive attention feature optimization based on the multi-scale feature map set to obtain an optimized feature map;

[0016] An analysis module, used for performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map;

[0017] A collaborative module, used for performing multi-label collaborative learning based on the semantic alignment feature map to obtain a multi-label classification result;

[0018] The recognition module is used to perform feature compression and recognition processing on the multi-label classification result to obtain a bill recognition result.

[0019] The third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned machine vision-based bill recognition method.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned machine vision-based bill recognition method.

[0021] In the technical solution provided by the present invention, by introducing random occlusion preprocessing and multi-scale feature extraction technology, the model's recognition ability for damaged and deformed bills is improved, and the robustness of the system is enhanced. The progressive attention feature optimization mechanism is adopted to achieve adaptive focusing on the key areas of the bill, and the discriminability and representation ability of the features are improved. The semantic alignment local feature analysis method is introduced to effectively integrate text semantic information with visual features, and the model's ability to understand the bill content is enhanced. Through the multi-label collaborative learning strategy, the correlation between different information fields in the bill is fully utilized to improve the accuracy and consistency of classification. The feature compression and efficient recognition processing technology are adopted to reduce the computational complexity, improve the processing efficiency of the system, and are suitable for large-scale bill recognition applications. The two-stage method combining template matching and fine recognition is used to achieve accurate recognition and field extraction of different types of bills, and has good generalization ability. Through the organic combination of multiple modules, an end-to-end bill recognition process is constructed, which simplifies the deployment and maintenance of the system. The present invention has strong scalability, and can adapt to new bill types by updating the model and template library to meet the dynamic needs in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0023] Figure 1 Schematic diagram of the steps of the bill recognition method based on machine vision in an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of the structure of a bill recognition device based on machine vision in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The embodiments of the present invention provide a machine vision-based bill recognition method and device. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the bill recognition method based on machine vision in the embodiment of the present invention includes:

[0027] Step S1, preprocessing and randomly blocking the input bill image to obtain a preprocessed bill image;

[0028] It is understandable that the execution subject of the present invention may be a bill recognition device based on machine vision, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0029] Specifically, the input bill image is size standardized, and the pixel size of the input image is adjusted to meet the preset standard size requirements to obtain a bill image of standard size. The bill image of standard size is converted into color space, and the image is converted into RGB color space. The bill image is converted into a color representation format commonly used in computer vision, so that the image data can better meet the needs of feature extraction algorithms in subsequent processing, ensuring that color information can be fully retained and effectively used for analysis. Based on the preset occlusion ratio range, a random number of rectangular occlusion areas are generated for the bill image in RGB color space. The generation of occlusion areas is achieved by randomly determining a number of rectangular areas on the image according to a preset ratio and range. In this process, the number and size of occlusion areas are random, and finally a set of occlusion areas is obtained, which simulates the damage or occlusion that may occur in the bill image in real scenes. Each rectangular area in the generated occlusion area set is randomly filled to simulate partial damage or contamination of the bill. The filling process uses random color or noise images to replace the pixel values ​​in these areas to form a simulated damage effect. After the filling is completed, the simulated damaged occluded area is superimposed on the original bill image in the RGB color space to obtain the bill image after occlusion processing. The contrast of the bill image after occlusion processing is adjusted to enhance the important features in the image, so that the light and dark contrast of the image is more obvious, which is convenient for subsequent feature extraction. After the contrast is adjusted, random noise is added to the contrast-enhanced image based on the preset noise parameters. This noise is usually generated in the form of Gaussian noise or salt and pepper noise to simulate the noise interference that may occur in the bill image in actual applications, and obtain a noisy bill image. The noisy bill image is edge enhanced to highlight the edge information in the image so that the machine vision system can more accurately identify the key features of the image in subsequent processing. Edge enhancement makes the edge details more obvious and improves the overall clarity of the image by performing specific filtering operations on the edge area of ​​the image, thereby obtaining a preprocessed bill image for subsequent recognition steps.

[0030] Step S2, performing multi-scale feature extraction on the pre-processed bill image to obtain a multi-scale feature map set;

[0031] Specifically, the preprocessed bill image is input into a deep convolutional neural network for feature extraction. The convolutional neural network extracts rich local feature information from the image through the operation of multiple convolutional layers. After the initial convolution operation, the initial feature map obtained contains the low-level and intermediate features of the bill image, which can describe the edge, texture and basic structure of the bill image. Based on the initial feature map, intermediate feature maps of different scales are generated through continuous convolution and pooling operations. The convolution operation extracts features from an image window of a specific size, while the pooling operation reduces the spatial resolution of the feature map by downsampling. The alternating convolution and pooling process not only reduces the computational complexity, but also can obtain image information at different scales. After multiple layers of convolution and pooling processing, multiple intermediate feature maps containing different details and scale information are generated. The intermediate feature maps of different scales are input into the feature pyramid network for processing. The feature pyramid network is a network structure that can effectively fuse features of different scales, and gradually fuses multi-level features through bottom-up feature extraction and top-down feature propagation. In the bottom-up process, the feature pyramid network performs a preliminary fusion of multi-scale feature maps to obtain a preliminary fused multi-scale feature map. Perform top-down feature propagation, starting from the high-level semantic features of the network, and gradually propagate these features to the lower layers, enhancing the semantic information in the high-level features so that these high-level features can better reflect the global semantic structure of the image. The enhanced high-level semantic features are upsampled to restore them to the same spatial resolution as the low-level feature maps. Upsampling enlarges the spatial size of the high-level semantic features through interpolation or deconvolution operations to keep them consistent with the size of the low-level feature maps. The upsampled high-level features are fused with the low-level features of the corresponding level. By combining the low-level features with rich details with the high-level features with strong semantics, a fused multi-scale feature map containing more details and global information is obtained. The non-local attention mechanism is applied to process the fused multi-scale feature map. The non-local attention mechanism introduces global context information, so that each feature point can refer to the information of other feature points and capture the long-distance dependencies in the image. Based on the feature map with global context information, feature reconstruction is performed through a multi-layer perceptron, and the features are refined and optimized to obtain the final multi-scale feature map set.

[0032] Step S3, performing progressive attention feature optimization based on a multi-scale feature map set to obtain an optimized feature map;

[0033] Specifically, the feature map with the lowest resolution in the multi-scale feature map set is grid-divided. By dividing the feature map with lower resolution into multiple grid regions, each grid can be independently processed to capture more details in the local area. Based on the initial grid feature map, the attention weight of each grid region is calculated by the convolutional neural network. The convolutional neural network extracts the features of each grid region and assigns different weight values ​​to these features to form an initial attention score map, which reflects the importance of each grid region in the entire image feature. The initial attention score map is softmax-normalized to normalize the attention scores of different grid regions into probability distributions to ensure that the sum of all attention weights is 1, so as to avoid a single grid region from having too much influence on other regions during the feature optimization process. The initial grid feature map is weighted. The features of each grid region are adjusted according to their corresponding attention weights. The higher the weight of the region, the greater the proportion of its features in the optimized feature map, and vice versa. Through the weighted optimization process, a weighted optimized feature map with higher expressiveness is obtained to better reflect the distribution of important information in the image. The weighted optimized feature map is upsampled to match its size to the feature map with higher resolution in the next layer. The upsampling operation expands the feature map from low resolution to the same size as the feature map of the next layer through interpolation or deconvolution technology, and can effectively fuse features with the original feature map of the layer. The feature fusion process combines the optimized features after upsampling with the original feature map of the next layer, so that the features of the two layers complement each other and improve the performance of the feature map. The optimization process is progressive, which means that it is iteratively executed layer by layer at different scales. In each layer of feature map, similar weighted optimization and upsampling processing are performed until the feature maps of all scales are optimized. Through this layer-by-layer optimization, multi-level optimized feature maps are gradually generated. The multi-level optimized feature map comprehensively considers the feature information of each scale and can fully reflect the information structure of the bill image at different scales. The multi-level optimized feature map is subjected to cross-scale feature aggregation. By integrating features of different scales, it is ensured that the features from low to high layers can complement and strengthen each other, forming a more complete feature expression, and obtaining an optimized feature map.

[0034] The initial grid feature map is subjected to channel compression processing to reduce the number of channels in the feature map, reduce the computational complexity, and retain the key feature information to obtain a more compact compressed feature map. Based on the compressed feature map, multi-scale spatial features are extracted through spatial pyramid pooling. Spatial pyramid pooling is a technology used to capture information of different scales of images. It extracts feature information at different scales by dividing the image into grids of different sizes and performing pooling operations on each grid. This process can effectively retain the global and local information of the image, so that the network can understand the structural characteristics of the bill image at multiple scales and obtain multi-scale spatial features. Inter-channel attention calculation is performed on the multi-scale spatial features. By calculating the weight of each channel, the relationship between different channels is captured, and which channels contain more important information is identified. The calculation of inter-channel attention is usually implemented through the self-attention mechanism, and the feature values ​​of each channel will be given different weights according to their importance. Through this process, a channel attention weight map is generated, which reflects the importance of different feature channels in the entire feature map. Based on the channel attention weights, weighted operations are performed on the multi-scale spatial features. The features of each channel are readjusted according to their corresponding attention weights. Channels with high weights occupy a larger proportion in the final feature expression, while channels with low weights are weakened. After the weighting operation, an optimized weighted spatial feature map is obtained. The weighted spatial feature map is deconvolved to restore its original size so that it is consistent with the spatial size of the initial grid feature map. Deconvolution is an upsampling technique that enlarges the resolution of the feature map through interpolation or the reverse operation of the convolution kernel so that the spatial structure of the feature map can be restored. Through the deconvolution operation, the obtained attention feature map has the same size as the initial grid feature map, which is convenient for subsequent local response calculation. Based on the attention feature map, the sliding window method is used to calculate the local response of each grid area. The sliding window technique calculates the response value of each local area step by step by moving a fixed-size window on the feature map to obtain a local response map of each grid area, reflecting the importance of each local area in the feature map. A nonlinear activation function is performed on the local response map, and each local response value is nonlinearly transformed using ReLU or other types of activation functions. The activation operation can enhance the effective feature response and suppress irrelevant noise information to obtain an activated attention response map. Based on the activated attention response map, a spatial softmax operation is performed. The softmax operation normalizes the attention values ​​in the response map to a probability distribution, ensuring that the sum of the attention values ​​of all local regions is 1, and obtaining the initial attention score map.

[0035] Step S4, performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map;

[0036] Specifically, optical character recognition is performed on the optimized feature map to extract text information from the bill image to form a continuous text sequence. The text sequence is input into a pre-trained semantic parsing model, which converts the text sequence into a text semantic vector by semantic encoding. The text semantic vector is processed by the self-attention mechanism to improve the expressiveness of semantic information. The self-attention mechanism captures the important semantic associations within the text by analyzing the relationship between semantic vectors and obtains an enhanced semantic representation. Based on the enhanced semantic representation, a semantic heat map is constructed to map the semantic information of the text back to the original image space. The function of the semantic heat map is to associate the semantic information with the specific area in the image and generate a semantic importance map. In order to improve the smoothness of the semantic information, the semantic importance map is Gaussian filtered to eliminate the noise in the image and make the semantic weight distribution more uniform, so as to obtain a smoothed semantic weight map. Based on the smoothed semantic weight map, the optimized feature map is weighted to enhance the areas in the feature map that are highly related to semantics, so that these areas occupy a more important position in the subsequent bill recognition, and generate a feature map with preliminary semantic alignment. The feature map with preliminary semantic alignment is input into the region proposal network for candidate region generation. The region proposal network can extract a series of local regions that may contain important information from the image. These regions are often the most recognizable and informative parts of the image. Through the region proposal network, a local region feature set is generated, which contains the feature representation of each important region in the image. Based on the local region feature set and enhanced semantic representation, the cross-modal attention mechanism is applied for feature fusion. The cross-modal attention mechanism captures the correlation between the image and text semantics by jointly processing the different modal information of the image and text semantics, so that the local region features in the image are aligned with the text semantic information at the semantic level, and a semantic alignment feature map is obtained.

[0037] The text semantic vector is calculated by multi-head self-attention. Through parallel calculation of multiple different attention heads, the model can focus on different parts of the text semantics from multiple perspectives. Each head generates an attention-weighted semantic vector by calculating the mutual relationship between the text semantic vectors. Based on the attention-weighted semantic vector, the feature transformation operation of the feedforward neural network is used to obtain a preliminary enhanced semantic representation. The preliminary enhanced semantic representation is processed by residual connection and layer normalization. The purpose of residual connection is to retain the original feature information of the input and fuse it with the enhanced features to avoid information loss or degradation. Layer normalization ensures the stability and consistency of the output of each layer by adjusting the feature distribution, and finally obtains an enhanced semantic representation. Based on this enhanced semantic representation, the correlation score of each semantic unit with the key information category of the bill is calculated to generate a semantic-category correlation matrix. The correlation matrix can reflect the degree of association between each semantic unit in the text and different categories in the bill (such as amount, date, invoice number, etc.), helping to identify which semantic information is closely related to specific key information of the bill. In order to extract important semantic information, the semantic-category correlation matrix is ​​subjected to a maximum pooling operation, and the most significantly related category is extracted from each semantic unit to obtain the category mapping of the semantic unit. Based on the category mapping information of the semantic unit and combined with the position information of the original image, a coarse-grained semantic heat map analysis is performed to obtain a preliminary semantic importance map. In order to match the resolution of the preliminary semantic importance map with the original image, the initial semantic importance map is upsampled to the same size as the original image using a bilinear interpolation method. Bilinear interpolation smoothly interpolates in space so that the heat map retains the original structure and information during the upsampling process, thereby obtaining an upsampled semantic importance map. The upsampled semantic importance map is segmented using an adaptive threshold. Through the adaptive threshold method, different thresholds are automatically determined based on the local information of the image, the most important semantic areas are segmented, and the parts of the image that are highly related to the text semantics are highlighted to obtain an accurate semantic importance map.

[0038] Step S5: Perform multi-label collaborative learning based on the semantic alignment feature map to obtain a multi-label classification result;

[0039] Specifically, the semantic alignment feature map is input into a multi-label classifier set. The classifier set consists of multiple independent classifiers, each of which corresponds to a different key information category in the bill, such as amount, date, invoice number, etc. Through the multi-label classifier set, an initial prediction result is generated for each category, representing the preliminary classification information of different categories in the bill. The initial prediction results are subjected to label correlation analysis to identify the possible dependencies between different labels, such as the inherent connection or co-occurrence pattern between certain bill information categories. The dependencies are quantified by constructing a label correlation matrix, and each element in the matrix represents the correlation strength between different labels. Based on the constructed label correlation matrix, the initial prediction results are optimized by conditional random fields. Conditional random fields are a probabilistic model for sequence labeling tasks. By combining the correlation between labels, the initial prediction results are globally optimized to improve the accuracy of classification. The prediction results optimized by conditional random fields can more fully reflect the dependencies between different category labels in the bill and correct unreasonable initial predictions. The optimized prediction results are input into the label smoothing module to reduce the risk of model overfitting. Through smoothing, the prediction results are made more robust and smooth, and not overly dependent on the accidental features in some training data. After smoothing, the confidence score of each label is calculated to generate a label confidence vector to quantify the confidence level of each label in the classification task. Based on the label confidence vector, dynamic threshold selection is performed to determine the optimal decision threshold for each label. Since the characteristics and confidence distribution of different labels may be different, dynamic threshold selection can set the most suitable decision criteria for each label to ensure the accuracy of the classification results. According to the label decision threshold set, the smoothed prediction results are binarized to generate the final target classification results. Binarization can convert the predicted continuous values ​​into a certain label category. Label conflict resolution and label completion are performed on the target classification results. In multi-label classification tasks, conflicts may occur between some labels, for example, some labels cannot exist at the same time logically. Label conflict resolution aims to resolve these contradictions and ensure the consistency of classification results. At the same time, label completion is used to supplement and improve some labels through reasoning or context information when some labels may not be predicted correctly, and finally obtain multi-label classification results.

[0040] Step S6: perform feature compression and recognition processing on the multi-label classification result to obtain the bill recognition result.

[0041] Specifically, feature encoding is performed on the multi-label classification results, and the multi-label classification results are converted into compressed feature vectors. Based on the predefined bill type template library, template matching operation is performed on the compressed feature vector. The template library stores a variety of different types of bill templates. By comparing the feature vector with the bill type template in the template library, the bill type candidate set that is most similar to the input bill is identified. Key field position information is extracted for each candidate type in the most similar bill type candidate set. By extracting the position of the key field from the candidate type, a candidate field position set is obtained, and the position information indicates the approximate position of the key field in each candidate bill in the image. Based on the candidate field position set, local feature extraction is performed on the input bill image. For each key field, the specific features of these fields are accurately identified from the input bill image to generate a local field feature set. The local field feature set is input into the pre-trained field recognition model for field recognition. The pre-trained field recognition model can identify each candidate field one by one and output a more accurate field recognition result set, which contains important field information in the bill and also reflects the specific content of these fields, such as invoice number, amount, date, etc. Based on the field recognition result set, calculate the comprehensive matching score of each candidate bill type. By comprehensively analyzing the correspondence between the field recognition results and each candidate bill type, generate a bill type matching score vector, which represents the degree of matching between each candidate bill type and the actual input bill. Perform weighted voting on the bill type matching score vector. In the weighted voting stage, select the optimal bill type as the final bill type recognition result based on the matching score of each candidate type. Weighted voting selects the type that best matches the actual input bill by comprehensively considering the matching scores of each candidate type. Combine the bill type recognition result with the field recognition result set to generate the final bill recognition result. By combining the bill type and the specific field content, output a complete bill parsing result, including the bill type and the content of each key field on it.

[0042] In the embodiment of the present invention, by introducing random occlusion preprocessing and multi-scale feature extraction technology, the model's recognition ability for damaged and deformed bills is improved, and the robustness of the system is enhanced. The progressive attention feature optimization mechanism is adopted to achieve adaptive focusing on the key areas of the bill, and the discriminability and representation ability of the features are improved. The semantic alignment local feature analysis method is introduced to effectively integrate text semantic information with visual features, and the model's ability to understand the bill content is enhanced. Through the multi-label collaborative learning strategy, the correlation between different information fields in the bill is fully utilized to improve the accuracy and consistency of classification. The feature compression and efficient recognition processing technology are adopted to reduce the computational complexity, improve the processing efficiency of the system, and are suitable for large-scale bill recognition applications. The two-stage method combining template matching and fine recognition is used to achieve accurate recognition and field extraction of different types of bills, and has good generalization ability. Through the organic combination of multiple modules, an end-to-end bill recognition process is constructed, which simplifies the deployment and maintenance of the system. The present invention has strong scalability, and can adapt to new bill types by updating the model and template library to meet the dynamic needs in practical applications.

[0043] In a specific embodiment, the process of executing step S1 may specifically include the following steps:

[0044] Performing size standardization processing on the input bill image to obtain a bill image of standard size, and performing color space conversion on the bill image of standard size to obtain a bill image in RGB color space;

[0045] Based on a preset occlusion ratio range, a random number of rectangular occlusion regions are generated for the bill image in the RGB color space to obtain an occlusion region set;

[0046] Perform random filling processing on each rectangular area in the occlusion area set to obtain a simulated damaged occlusion area, and superimpose the simulated damaged occlusion area on the bill image in the RGB color space to obtain the bill image after occlusion processing;

[0047] Adjusting the contrast of the bill image after the occlusion processing to obtain a bill image with enhanced contrast, and adding random noise to the bill image with enhanced contrast based on a preset noise parameter to obtain a bill image with noise;

[0048] The edge of the noisy bill image is enhanced to obtain the preprocessed bill image.

[0049] Specifically, the bill image is normalized by adjusting the size of the image to conform to a preset standard size. For example, all input images are scaled to a specific width and height. ,in, represents the normalized image width, Represents the standardized image height. This process can be achieved through bilinear interpolation or nearest neighbor interpolation, the formula is:

[0050] ;

[0051] in, represents the normalized pixel value, is the pixel value of the original image, is the pixel coordinate in the normalized image, is the pixel coordinate in the original image. Through this operation, a bill image of standard size is obtained. Perform color space conversion on the bill image of standard size and convert the image into RGB color space. RGB color space represents each pixel value in the image through three color channels: red (R), green (G) and blue (B). This method can retain the color information of the image to the maximum extent. Based on the preset occlusion ratio range, a random number of rectangular occlusion areas are generated for the bill image in RGB color space. The occlusion area is used to simulate the damage or occlusion that the bill may encounter in actual use. Set an occlusion ratio range according to the input image size For example, if the occlusion ratio is set between 5% and 20%, a different number of rectangular occlusion areas will be randomly generated. The size and position of each occlusion area are also determined randomly. The width and height of the occlusion area can be set within a certain ratio range. Assuming the width of the image is , the height is , then the width of each occluded area and height The following ranges can be randomly selected:

[0052] ;

[0053] in, and is a randomly selected scaling factor, usually randomly selected between 0.05 and 0.2. This operation obtains a set of occluded regions, indicating which parts of the image will be occluded. The occluded regions are randomly filled to simulate the situation of damaged bills. The random filling content can be noise, random color blocks, or other pixel values ​​that are unrelated to the image content. For example, the occluded region is filled using a Gaussian noise model. Gaussian noise can be expressed by the following formula:

[0054] ;

[0055] in, is the pixel value after filling noise, is the pixel value of the original image, The mean is 0 and the variance is Gaussian noise. Through this operation, a simulated damaged occlusion area is generated. The occlusion area is superimposed on the bill image in the RGB color space to obtain the bill image after occlusion processing. The contrast of the bill image after occlusion processing is adjusted to enhance the difference between the bright and dark areas in the image, making the key features more obvious. Contrast adjustment can be achieved through linear transformation. Assuming that the image pixel value is in the range [0, 255], it can be adjusted by the following formula:

[0056] ;

[0057] in, is the pixel value after contrast enhancement, is the contrast enhancement factor, and a value greater than 1 (such as 1.2 or 1.5) is usually selected to increase the contrast of the image. After contrast adjustment, the features of the image are clearer, which is convenient for subsequent recognition. In order to increase the robustness of the image, random noise is added to the contrast-enhanced image based on the preset noise parameters to simulate the signal noise or image quality problems that may occur during image acquisition in the real world. The type of noise can be Gaussian noise or salt and pepper noise. By adding these random noises, a noisy bill image is generated. The edge of the noisy bill image is enhanced to highlight the detailed information of the edge area in the image, making the key features in the bill (such as text and lines) more obvious. Edge enhancement is usually performed using the Laplace operator or the Sobel operator. Taking the Laplace operator as an example, its calculation formula is:

[0058] ;

[0059] in, is the pixel value after edge enhancement, is the Laplace operator of the image. Through this operation, the preprocessed bill image is obtained.

[0060] In a specific embodiment, the process of executing step S2 may specifically include the following steps:

[0061] The preprocessed bill image is input into the deep convolutional neural network for feature extraction to obtain the initial feature map. Based on the initial feature map, multiple intermediate feature maps of different scales are obtained through continuous convolution and pooling operations.

[0062] Perform feature pyramid network processing on multiple intermediate feature maps of different scales to obtain a preliminary fused multi-scale feature map, and perform top-down feature propagation based on the preliminary fused multi-scale feature map to obtain enhanced high-level semantic features;

[0063] The enhanced high-level semantic features are upsampled to obtain an upsampled feature map with the same size as the low-level feature map, and the upsampled feature map is fused with the low-level feature map of the corresponding level to obtain a fused multi-scale feature map;

[0064] A non-local attention mechanism is performed on the fused multi-scale feature map to obtain a feature map with global context information. Based on the feature map with global context information, feature reconstruction is performed through a multi-layer perceptron to obtain a set of multi-scale feature maps.

[0065] Specifically, the preprocessed bill image is input into a deep convolutional neural network for feature extraction, and the key features in the image are extracted to obtain an initial feature map. The convolutional neural network processes the image through multiple convolutional layers and nonlinear activation functions (such as ReLU), which can effectively capture local features in the image, such as edges, lines, and textures. The convolution operation can be expressed as:

[0066] ;

[0067] in, represents the feature map after convolution, is the pixel value of the input image, is the convolution kernel. The convolution kernel slides on the image to generate an initial feature map, which describes the features of different scales in the image, such as text, numbers, lines on the bill, etc. Based on the initial feature map, multiple intermediate feature maps of different scales are obtained through continuous convolution and pooling operations. The pooling operation reduces the size of the feature map by downsampling while retaining the most important information. The pooling formula is as follows:

[0068] ;

[0069] in, represents the feature map after pooling, The function represents selecting the maximum value within the window, thereby effectively reducing the resolution of the feature map, reducing the amount of calculation, and helping the network to better capture the high-level semantic information of the image. Through multi-layer convolution and pooling operations, a series of intermediate feature maps of different scales are generated, covering feature information of different levels and layers of the image. The intermediate feature map can reflect the multi-level features from low-level pixel information to high-level semantic information. The feature pyramid network is used to process multiple intermediate feature maps of different scales, and the feature maps of different scales are fused together to form a more expressive multi-scale feature map. The feature pyramid network adopts a top-down structure. On each layer of the feature pyramid, high-level semantic features are gradually propagated to low-level features through layer-by-layer upsampling and feature fusion. The top-down feature propagation restores the high-level feature map to the resolution of the low-level feature map through upsampling operations. The upsampling process can be achieved through bilinear interpolation or deconvolution, and the formula is:

[0070] ;

[0071] in, is the upsampled feature map, is the original feature map, is the upsampling weight. Through upsampling, the resolution of high-level semantic features is kept consistent with the low-level feature map, and they can be fused at the same scale. During the fusion process, the high-level semantic features obtained by upsampling are fused with the low-level feature maps of the corresponding level to obtain a fused multi-scale feature map. For example, the low-level feature map may contain the text edge information in the bill, while the high-level features may contain the overall bill layout structure. Through fusion, these two types of information can be used together for bill recognition tasks. The fused feature map is processed by the non-local attention mechanism, and the long-distance dependency in the image is captured by calculating the feature correlation between any two positions in the image. Usually, the convolution operation only focuses on local information, while the non-local attention mechanism can capture contextual information on a global scale. The non-local operation can be expressed as:

[0072] ;

[0073] in, is the output feature, and is the pixel point of the input feature map, is the similarity function, usually calculated by dot product or Gaussian function, is the input feature map, C is the normalization factor. Through the non-local attention mechanism, the network can better understand the global structure and context information of the bill image and improve the global expression ability of the feature map. After obtaining the feature map with global context information, the multi-layer perceptron (MLP) is used to reconstruct the features. MLP further transforms the fused multi-scale feature map through multiple fully connected layers and nonlinear activation functions to generate a more abstract feature expression. The basic formula of MLP is:

[0074] ;

[0075] in, is the output feature, is the input feature, is the weight matrix, For the accessories, is an activation function (such as ReLU). Through this operation, the features are recombined and transformed, so that the expressive power of the feature map is further enhanced. After reconstruction by the multi-layer perceptron, the final feature map set contains multi-scale and multi-level features. These features not only contain the details and structural information of the bill image, but also capture the global context information through top-down feature propagation and non-local attention mechanism.

[0076] In a specific embodiment, the process of executing step S3 may specifically include the following steps:

[0077] Gridding the feature map with the lowest resolution in the multi-scale feature map set to obtain an initial grid feature map;

[0078] Based on the initial grid feature map, the attention weight of each grid area is calculated through a convolutional neural network to obtain the initial attention score map;

[0079] The initial attention score map is subjected to softmax normalization to obtain a normalized attention weight map, and based on the normalized attention weight map, the initial grid feature map is subjected to weighted processing to obtain a weighted optimized feature map;

[0080] The weighted optimized feature map is upsampled to the size of the feature map of the next layer to obtain the target feature map, and the target feature map is fused with the original feature map of the next layer to obtain a fused feature map;

[0081] Iterative execution is performed based on the fused feature map until feature maps of all scales are processed to obtain a multi-level optimized feature map, and cross-scale feature aggregation is performed on the multi-level optimized feature map to obtain an optimized feature map.

[0082] Specifically, the feature map with the lowest resolution is selected from the multi-scale feature map set. The feature map with the lowest resolution is usually generated by the pooling operation in the deep convolutional neural network. The pooling operation will gradually reduce the spatial resolution of the image while retaining important high-level semantic information. The low-resolution feature map has fewer spatial details but contains rich semantic information. The feature map with the lowest resolution is divided into grids. Assume that the size of the feature map is , divide it into multiple fixed-size grid areas. The grid size can be set according to task requirements. For example, the size of each grid is , then the feature map can be divided into Grid regions. These grids can extract local features respectively. After the grid division is completed, a convolutional neural network is used to calculate each grid region to extract the attention weight. For each grid region, the convolutional neural network can extract the high-order features in the region through convolution operations and generate a corresponding attention score for the region. The convolution operation of the convolutional neural network can be expressed by the following formula:

[0083] ;

[0084] in, represents the attention score of the grid area, is the eigenvalue in the initial grid feature map, is a convolution kernel used to extract local features. Through the convolution operation, the initial attention score map is obtained, which contains the importance weight of each grid area. The initial attention score map is normalized by softmax to convert the attention score into a probability distribution, ensuring that the attention weight of each grid area is between 0 and 1, and the sum of the weights of all grids is 1. The formula for the softmax operation is as follows:

[0085] ;

[0086] in, It is The normalized attention weights of the grid, is the initial attention score, Represents exponential operation. After softmax processing, a normalized attention weight map is obtained, which reflects the relative importance of each grid area in the entire feature map. Based on the normalized attention weight map, the initial grid feature map is weighted. The attention weight of each grid is used to weight the features of the corresponding grid to improve the feature representation ability of important grid areas and reduce the influence of unimportant areas. Weighted processing can be achieved by the following formula:

[0087] ;

[0088] in, is the weighted optimized feature, is the normalized attention weight, is the initial grid feature. The weighted feature map highlights those areas with higher attention weights, so that these areas play a greater role in subsequent feature extraction, thereby improving the accuracy of recognition. The weighted optimized feature map is upsampled to the size of the next layer feature map. Since the resolution of the previous layer feature map is low and the resolution of the next layer feature map is high, the purpose of upsampling is to make the optimized feature map have the same size as the next layer feature map to facilitate feature fusion between them. Upsampling can be achieved by bilinear interpolation, and the formula is as follows:

[0089] ;

[0090] in, is the upsampled feature map, is the weighted optimized feature map, is the interpolation weight. By upsampling, a target feature map with the same size as the feature map of the next layer is obtained. The upsampled feature map is fused with the feature map of the next layer, and the information of the two layers of feature maps is combined by pixel-by-pixel addition to obtain a fused feature map containing multi-layer feature information. The above steps are an iterative process. After each layer of feature fusion is completed, the result is used as the input of the next layer, and upsampling and fusion are continued until feature maps of all scales are processed. The bottom-up multi-level feature optimization process enables the feature map to contain both low-level detail information and high-level semantic information at each layer, and gradually obtains a multi-level optimized feature map. Cross-scale feature aggregation is performed on the multi-level optimized feature map, and features of different scales are further fused to ensure that information of all scales can be expressed in the final feature map. Through feature aggregation, a unified optimized feature map is generated.

[0091] In a specific embodiment, the execution step is based on the initial grid feature map, and the attention weight of each grid area is calculated by a convolutional neural network to obtain the initial attention score map. The process may specifically include the following steps:

[0092] Perform channel compression processing on the initial grid feature map to obtain a compressed feature map, and based on the compressed feature map, perform spatial pyramid pooling operation to obtain multi-scale spatial features;

[0093] Perform inter-channel attention calculation on the multi-scale spatial features to obtain the channel attention weights, and weight the multi-scale spatial features based on the channel attention weights to obtain weighted spatial features;

[0094] Deconvolution operation is performed on the weighted spatial features to obtain an attention feature map with the same size as the initial grid feature map. Based on the attention feature map, the local response of each grid area is calculated by the sliding window method to obtain a local response map.

[0095] A nonlinear activation function is performed on the local response map to obtain the activated attention response map, and based on the activated attention response map, a spatial softmax operation is performed to obtain the initial attention score map.

[0096] Specifically, the high-dimensional initial grid feature map is reduced in dimensionality to ensure its computational efficiency is improved while retaining key features. The initial grid feature map usually contains multiple channels, each channel corresponding to a different feature map. Through channel compression processing, the number of channels of the feature map is reduced, and the high-dimensional features are compressed into a lower-dimensional representation. The common method of channel compression is to achieve it through 1x1 convolution, and its formula is as follows:

[0097] ;

[0098] in, is the compressed feature map, is the first Channel feature maps, is the weight of the 1x1 convolution kernel. Through 1x1 convolution, the information of multiple channels is integrated into a few channels to achieve the effect of dimensionality reduction, reduce the computational complexity of subsequent processing, and retain the most important feature information. Based on the compressed feature map, spatial pyramid pooling operation is performed to generate multi-scale spatial features. Spatial pyramid pooling is an effective multi-scale feature extraction method. It extracts multi-scale spatial features by dividing the feature map into regions of different scales and performing pooling operations in each region. Assuming that the compressed feature map is divided into different pooling windows, for example, windows of sizes such as 1x1, 2x2, 4x4, the formula for spatial pyramid pooling is:

[0099] ;

[0100] in, Indicated in scale The pooling operation on It is a scale The feature representation after pooling is is a compressed feature map. By performing pooling operations at multiple scales, spatial pyramid pooling can capture spatial information of different scales. Multi-scale features can provide rich spatial information in the subsequent recognition process, allowing the model to better handle complex visual tasks. Inter-channel attention calculation is performed on multi-scale spatial features to capture the dependencies and importance between different channels of the feature map. By assigning different attention weights, the contribution of important channels is enhanced and the influence of irrelevant or redundant channels is suppressed. The calculation of inter-channel attention can be achieved through the self-attention mechanism. The specific formula is:

[0101] ;

[0102] in, It is The attention weight of each channel, Represents the first channels, is the importance function of the feature channel, which can usually be implemented through a fully connected layer or convolution operation. Through attention calculation, it automatically identifies which channels are more important for a specific task and adjusts their weights accordingly. Based on the calculated channel attention weights, the multi-scale spatial features are weighted. The feature value of each channel is multiplied by its corresponding attention weight to obtain the weighted spatial feature. This operation can be expressed by the following formula:

[0103] ;

[0104] in, is the weighted Channel features, is the attention weight of the channel, is the first After weighting, important channel features are amplified, while unimportant channel features are weakened, so that the model pays more attention to key features in subsequent processing. Deconvolution is performed on the weighted spatial features to restore them to the same size as the initial grid feature map. Deconvolution (or upsampling) is used to enlarge the low-resolution feature map to the same size as the original feature map. Figure 1 The deconvolution operation can be achieved by back propagation of the convolution kernel. The specific formula is:

[0105] ;

[0106] in, is the feature map after deconvolution, is the weighted spatial feature map, is the weight of deconvolution. Through deconvolution, the resolution of the feature map is improved so that it can be effectively aligned with the original grid feature map. After obtaining the attention feature map with the same size as the initial grid feature map, the local response of each grid area is calculated based on these feature maps using the sliding window method. The sliding window method can extract the response value of each local area by sliding a fixed-size window on the feature map and performing weighted summation of the features in the window. This method can be expressed by the following formula:

[0107] ;

[0108] in, is the local response within the sliding window, is the feature value in the attention feature map, is the weight of the sliding window. Through the sliding window method, the importance of each local area in the feature map is captured to generate a local response map. A nonlinear activation function is performed on the local response map. A nonlinear activation function such as ReLU (linear correction function) is used to process the local response to enhance the expressiveness of the feature. The formula of ReLU is:

[0109] ;

[0110] in, is the local response after activation. ReLU sets negative values ​​to 0 and retains positive values, thereby ensuring that negative values ​​in the feature map do not affect the final calculation results. Based on the activated attention response map, a spatial softmax operation is performed to generate the initial attention score map. The softmax operation is used to convert the attention response into a probability distribution, ensuring that the response value of each local area is between 0 and 1, and the sum of the response values ​​of all areas is 1. The formula for softmax is:

[0111] ;

[0112] in, is the attention value in the initial attention score map, is the local response after activation. Through softmax normalization, the obtained attention score map can provide a more accurate feature weight distribution for subsequent models.

[0113] In a specific embodiment, the process of executing step S4 may specifically include the following steps:

[0114] Perform optical character recognition on the optimized feature graph to extract text information from the bill to obtain a text sequence. Then, based on the pre-trained semantic parsing model, perform semantic encoding on the text sequence to obtain a text semantic vector.

[0115] The text semantic vector is processed by the self-attention mechanism to obtain an enhanced semantic representation. Based on the enhanced semantic representation, a semantic heat map is constructed to map the semantic information back to the original image space to obtain a semantic importance map.

[0116] Perform Gaussian filtering on the semantic importance map to obtain a smoothed semantic weight map, and based on the smoothed semantic weight map, weight the optimized feature map to obtain a feature map with preliminary semantic alignment;

[0117] The preliminary semantically aligned feature map is input into the region proposal network to generate candidate regions to obtain a local region feature set. Based on the local region feature set and the enhanced semantic representation, feature fusion is performed through a cross-modal attention mechanism to obtain a semantically aligned feature map.

[0118] Specifically, the optimized feature map is input into the optical character recognition (OCR) engine. OCR technology can extract key information from the bill, such as invoice number, amount, date and other text information by recognizing characters in the image. The OCR system usually includes multiple steps such as image preprocessing, character segmentation and character recognition. In the image preprocessing stage, the optimized feature map undergoes denoising, binarization and other operations to make the characters in the image clearer. Character segmentation separates the characters one by one by analyzing the text boundaries in the image. The character recognition part converts the characters in the image into a text sequence by comparing the shape features of the characters. The text sequence is semantically encoded using a pre-trained semantic parsing model. The semantic parsing model can convert the text sequence into a semantic vector through natural language processing technology. Commonly used semantic encoding models include BERT, GPT, etc., which can convert the contextual information in the text sequence into a high-dimensional vector representation. Text sequence Each word or character in is encoded into a vector through the semantic model , and finally get the semantic vector of the text ,in Indicates The semantic vector of a character or word. The semantic vector contains not only the surface information of the character but also its meaning in the context. The self-attention mechanism is used to process the text semantic vector. The self-attention mechanism captures the long-distance dependency and context information in the text by calculating the relationship between different semantic vectors. The calculation of self-attention can be expressed by the following formula:

[0119] ;

[0120] in, Indicates The semantic vector and The attention weights between semantic vectors, is the dot product between semantic vectors. Through the self-attention mechanism, the model can enhance the representation ability of key semantic information, reduce the weight of relatively unimportant parts, and obtain a more accurate semantic representation. Based on the enhanced semantic representation, a semantic heat map is constructed. The semantic heat map maps the semantic information in the text back to the original image space of the bill to show which image areas are most relevant to specific semantic information. By mapping the position of each semantic vector in the image, a semantic importance map is generated to show which areas play a key role in the text content of the bill. For example, in the bill, the area where the invoice number is located is usually mapped as a high-weight area, while other areas have lower weights. The semantic heat map reflects the correspondence between text information and image space, which can better perform subsequent processing. In order to make the semantic heat map smoother and more coherent, the semantic importance map is Gaussian filtered. Gaussian filtering smoothes the image through convolution operations, which can eliminate noise and discontinuity and make the semantic weight more uniform in space. The formula for Gaussian filtering is:

[0121] ;

[0122] in, is a Gaussian filter, is the standard deviation of the filter. By applying Gaussian filtering, the noise area in the semantic weight map is suppressed, while the features of the key areas are retained to form a smoothed semantic weight map. Based on the smoothed semantic weight map, the optimized feature map is weighted to highlight the important areas related to the semantic information and weaken the influence of irrelevant areas. The weighted operation can be expressed by the following formula:

[0123] ;

[0124] in, is the weighted feature map, is the smoothed semantic weight, is the optimized feature map. In the weighting process, the features of the key areas in the bill are highlighted according to the semantic information. The feature map with preliminary semantic alignment is input into the region proposal network for candidate region generation. The region proposal network automatically generates a series of regions that may contain key information from the image, which usually include text, fields, barcodes, etc. in the bill. The region proposal network traverses the feature map through a sliding window and generates a bounding box and score of a candidate region for each window. The region proposal network selects the region most likely to contain key information based on these scores and generates a candidate region feature set. After obtaining the local region feature set, the cross-modal attention mechanism is used for feature fusion based on the local region features and enhanced semantic representation. The cross-modal attention mechanism effectively combines the information of the two by analyzing the relationship between image features and semantic features. For example, by calculating the attention weights of local region features and semantic vectors, it is possible to identify which image regions are associated with semantic information and adjust the weights of these regions accordingly. The calculation formula of cross-modal attention is as follows:

[0125] ;

[0126] in, is the image feature and semantic features The attention weight between them. Through fusion, the model can use both image and text information to improve the accuracy and robustness of recognition. The resulting semantically aligned feature map combines the visual features of the image and the semantic information of the text.

[0127] In a specific embodiment, the execution step performs a self-attention mechanism on the text semantic vector to obtain an enhanced semantic representation, and based on the enhanced semantic representation, a semantic heat map is constructed to map the semantic information back to the original image space. The process of obtaining the semantic importance map can specifically include the following steps:

[0128] Perform multi-head self-attention calculation on the text semantic vector to obtain the attention-weighted semantic vector, and then perform feature transformation through a feedforward neural network based on the attention-weighted semantic vector to obtain a preliminary enhanced semantic representation;

[0129] Perform residual connection and layer normalization on the preliminary enhanced semantic representation to obtain enhanced semantic representation, and based on the enhanced semantic representation, calculate the relevance score of each semantic unit with the key information category of the bill to obtain the semantic-category relevance matrix;

[0130] Perform a maximum pooling operation on the semantic-category correlation matrix to extract the most significant category of each semantic unit, obtain the semantic unit category map, and perform a coarse-grained semantic heat map analysis based on the semantic unit category map and the position information of the original image to obtain the initial semantic importance map;

[0131] Perform bilinear interpolation on the initial semantic importance map, upsample the initial semantic importance map to the same size as the original image, and obtain an upsampled semantic importance map;

[0132] Adaptive threshold segmentation is performed on the upsampled semantic importance map to obtain the semantic importance map.

[0133] Specifically, multi-head self-attention calculation is performed on the text semantic vector. Each semantic vector represents a word or character in the text sequence and embeds its context information into a high-dimensional vector space. The multi-head self-attention mechanism enhances the model's understanding of long-distance dependencies and different contexts by calculating different attention distributions in parallel on multiple independent attention heads. The multi-head self-attention mechanism can be expressed by the following formula:

[0134] ;

[0135] in, , Represent the query, key and value matrices respectively. These parameters are transformed linearly on the input semantic vector. is the dimension of the key vector, which is used for normalization to prevent the inner product from being too large. Through the parallel calculation of multiple attention heads, the model can analyze the relationship between semantic vectors from different angles and generate multi-head attention weighted semantic vectors. Based on the weighted semantic vector, feature transformation is performed through a feedforward neural network. The feedforward neural network consists of one or more fully connected layers, each of which is followed by a nonlinear activation function (such as ReLU) to perform linear transformation and nonlinear processing on the input weighted semantic vector to enhance the representation ability of the model. The calculation formula of the feedforward network is:

[0136] ;

[0137] in, is the weighted semantic vector, and is the weight matrix, and is the bias term, Represents the ReLU activation function. After being processed by the feedforward neural network, the obtained semantic representation contains the contextual information after deep feature transformation, forming a preliminary enhanced semantic representation. In order to ensure that the model maintains the consistency of input information at multiple levels, the preliminary enhanced semantic representation is subjected to residual connection and layer normalization. Residual connection skips certain transformation steps and directly adds the input information to the output, thereby avoiding the gradual attenuation or loss of information in the deep network. The formula is as follows:

[0138] ;

[0139] Among them, LayerNorm(·) represents layer normalization, FFN is the semantic vector after the feedforward network, is the original input. Layer normalization is used to standardize the distribution of each semantic vector to ensure that the values ​​on different dimensions are in the same range, which helps to accelerate the convergence of the model. Based on the enhanced semantic representation, the relevance score of each semantic unit with the key information category of the bill is calculated. This is achieved by calculating the similarity between the semantic vector and different categories of bills (such as invoice number, amount, date, etc.). Using the inner product or similarity calculation formula, the semantic vector is compared with the embedding vector of each category to obtain the semantic-category correlation matrix. The formula is:

[0140] ;

[0141] in, Indicates The semantic unit and The relevance score of each bill category, It is The vector of semantic units, It is The correlation matrix can reveal the degree of correlation between different semantic information in the text and various key information of the bill. In order to extract the most significant category of each semantic unit, the semantic-category correlation matrix is ​​subjected to the maximum pooling operation, and the category with the highest score is extracted from the correlation of each semantic unit, thereby effectively filtering out the most important category represented by each semantic unit. The pooling operation can be expressed by the following formula:

[0142] ;

[0143] in, Indicates The most significant category score of the semantic units is obtained by taking the maximum value of each row in the correlation matrix to obtain the semantic unit category mapping. Based on these mappings, each semantic unit is mapped to the key information category in the bill to generate a category label sequence. Based on the semantic unit category mapping and the position information of the original image, a coarse-grained semantic heat map analysis is performed. By mapping the semantic categories to the spatial positions of the bill image, an initial semantic importance map is generated. This map can show which areas in the image are more closely associated with important categories in the text, for example, the invoice number may correspond to a specific location, and the amount may be located in the upper right corner of the bill. This process can combine text information and image features. The initial semantic importance map is bilinearly interpolated and upsampled to the same size as the original image. The formula for bilinear interpolation is:

[0144] ;

[0145] in, is the interpolated pixel value, is the pixel value in the initial semantic importance map. Through weight distribution, the low-resolution image is enlarged to match the original image size, so that the semantic information can be accurately mapped to the specific area of ​​the bill image. In the upsampling process, the upsampled semantic importance map is adaptively thresholded. According to the different semantic weights, the threshold of each area is automatically determined, the areas with higher semantic weights are retained, and irrelevant areas are filtered out. The formula for adaptive threshold segmentation is as follows:

[0146] ;

[0147] in, is the segmented image, The adaptive threshold is dynamically calculated according to the local features of the image to ensure that the segmentation result of each region is more reasonable. Through this process, the final semantic importance map is generated to highlight the areas related to key information in the bill and provide reliable feature support for the bill recognition task.

[0148] In a specific embodiment, the process of executing step S5 may specifically include the following steps:

[0149] Inputting the semantic alignment feature map into a multi-label classifier set to obtain an initial prediction result, where the multi-label classifier set includes multiple independent classifiers corresponding to the key information categories of the bill;

[0150] Perform label correlation analysis on the initial prediction results and construct a label correlation matrix, which represents the dependency relationship between different labels.

[0151] Based on the label correlation matrix, the initial prediction results are optimized by conditional random field optimization to obtain optimized prediction results;

[0152] The optimized prediction results are input into the label smoothing module, and the overfitting risk is reduced by label smoothing technology to obtain smoothed prediction results. Based on the smoothed prediction results, the confidence score of each label is calculated to obtain the label confidence vector.

[0153] Dynamic threshold selection is performed on the label confidence vector to determine the optimal decision threshold for each label, and a label decision threshold set is obtained. Based on the label decision threshold set, the smoothed prediction result is binarized to obtain the target classification result.

[0154] Perform label conflict resolution and label completion processing on the target classification results to obtain multi-label classification results.

[0155] Specifically, multiple independent classifiers are designed for different categories of key bill information (such as invoice number, amount, date, etc.). These classifiers can use different model structures (such as support vector machines, decision trees or neural networks, etc.), and each classifier specializes in processing the features of one category, thus forming a multi-label classifier set. Given the semantic alignment feature map of the input , each independent classifier will extract features related to its target label and output the corresponding prediction result. Assume that the output of the classifier is , then the initial prediction result is expressed as ,in It is The classifier is The prediction of each category is performed. The label correlation analysis is performed on the initial prediction results. Since different labels in multi-label classification tasks are often not independent, for example, the invoice number and date in the bill may exist at the same time or not, the dependency relationship between labels is modeled. By analyzing the label co-occurrence in the initial prediction results, a label correlation matrix is ​​constructed. Each element of the label correlation matrix represents the correlation between two labels. The calculation formula of the matrix is:

[0156] ;

[0157] in, Representation Tags and tags The correlation between is the covariance between the two, and Var The labels are and tags The variance of . Through this matrix, we can identify which labels have strong correlations. Based on the label correlation matrix, the initial prediction results are optimized using the conditional random field. The conditional random field is a probabilistic graphical model used to model sequence data or multi-label data. It can take into account the dependencies between labels, thereby globally optimizing the initial prediction results. The goal of the conditional random field is to maximize the joint probability distribution of the labels. The optimization process can be expressed by the following formula:

[0158] ;

[0159] in, is the optimized prediction probability, and are model parameters, Representation Tags and tags The relevance of is the characteristic function, is the normalization constant. Through conditional random field optimization, the interdependence between labels is fully utilized to generate more consistent and accurate label prediction results. The optimized prediction results are input into the label smoothing module, and the model is prevented from overfitting through regularization technology. Common label smoothing methods include fusing the predicted probability distribution with the uniform distribution, thereby reducing the model's dependence on specific patterns in the training data and ensuring that the model has better generalization ability for new data. The prediction result after label smoothing can be expressed as:

[0160] ;

[0161] in, is the smoothed predicted probability, is the smoothing coefficient, is the number of categories. By smoothing the labels, the risk of over-preference of some labels is reduced, making the prediction results more stable and robust. Based on the smoothed prediction results, the confidence score of each label is calculated to generate a label confidence vector. The confidence score reflects the probability strength of each label being predicted as a certain category, which can help the system make more reasonable decisions in subsequent steps. The confidence score can be directly generated based on the predicted probability, and the vector form is:

[0162] ;

[0163] in, is the label confidence vector, which contains the confidence of each label. Dynamic threshold selection is performed on the label confidence vector to determine the optimal decision threshold for each label. The dynamic threshold selection method dynamically adjusts the decision threshold according to the confidence distribution of each label. The optimal threshold is selected for each label through cross-validation or statistical analysis. Assume that the decision threshold of the label is , then the threshold set is , and based on this threshold set, the smoothed prediction results are binarized to generate the target classification results:

[0164] ;

[0165] in, It is the label prediction result after binarization, 1 means the label exists, and 0 means the label does not exist. Through the dynamic threshold method, the system can handle the prediction of different category labels more flexibly. Perform label conflict resolution and label completion processing on the target classification result. The purpose of label conflict resolution is to solve the problem of logical conflicts between certain labels, such as certain category labels cannot appear at the same time. These conflicts are identified and resolved by referring to the label correlation matrix. At the same time, label completion is used to complete certain labels based on contextual information or the prediction results of other labels when they are not predicted correctly. For example, if some labels are closely related semantically and one of them is predicted correctly, the system can infer the existence of related labels. After conflict resolution and label completion, the multi-label classification result is obtained.

[0166] In a specific embodiment, the process of executing step S6 may specifically include the following steps:

[0167] Perform feature encoding on the multi-label classification results to obtain a compressed feature vector, and perform template matching on the compressed feature vector based on a predefined bill type template library to obtain the most similar bill type candidate set;

[0168] Extract key field position information of each candidate type in the most similar bill type candidate set to obtain a candidate field position set, and extract local features of the input bill image based on the candidate field position set to obtain a local field feature set;

[0169] Input the local field feature set into the pre-trained field recognition model, perform fine recognition on each candidate field, obtain a field recognition result set, and calculate the comprehensive matching score of each candidate bill type based on the field recognition result set to obtain a bill type matching score vector;

[0170] A weighted vote is performed on the bill type matching score vector to select the final bill type, obtain the bill type recognition result, and combine the bill type recognition result with the field recognition result set to generate the bill recognition result.

[0171] Specifically, the multi-label classification results are compressed to obtain a compact feature vector representation. Each label represents a key information category on the bill, such as invoice number, amount, date, etc. The initial representation of the label is a high-dimensional binary or probability distribution vector. Feature encoding converts the high-dimensional multi-label classification results into a low-dimensional feature vector through dimensionality reduction technology, which is achieved through principal component analysis, autoencoder or sparse coding. Assume that the multi-label classification result is , where each Representative The classification result of labels, the compressed feature vector can be expressed as:

[0172] ;

[0173] in, is the compressed feature vector, It is a dimensionality reduction matrix or a weight matrix, which is obtained through learning. It compresses the high-dimensional multi-label representation into a low-dimensional space. The compressed feature vector retains the representation of the key information in the bill. Based on the predefined bill type template library, the compressed feature vector is template matched. The bill type template library contains standard feature vectors of multiple different bill types, and each bill type has a unique label combination and field features. By matching the input compressed feature vector with the template in the template library, the type candidate set that is most similar to the input bill is found. The matching process can measure the similarity between feature vectors by cosine similarity or Euclidean distance. For example, the formula for cosine similarity is:

[0174] ;

[0175] in, Indicates The feature vector of the template, Represents the input feature vector With template The similarity between Represents the norm of the vector. By calculating the similarity between the input feature vector and each template, the templates with the highest similarity are selected as the candidate set of bill types. After obtaining the most similar candidate set of bill types, the key field position information is extracted for each candidate type. According to the template of the candidate bill type, the position of the key field corresponding to the bill type in the image is extracted. These fields usually include invoice number, amount, date, etc., which may be located in different areas on different types of bills. By using pre-defined templates, the field position of each candidate bill type is identified and a candidate field position set is generated. Based on the candidate field position set, local features are extracted from the input bill image. Corresponding local features are extracted from each key field area of ​​the bill image. These features may include information such as the edge, texture, and color of the image. By extracting local features, detailed information related to the field in the bill image is obtained to obtain a local field feature set. These local features can effectively capture the visual information of different fields in the bill and support subsequent field recognition tasks. The local field feature set is input into the pre-trained field recognition model to perform fine recognition on each candidate field. Field recognition models are usually based on deep learning, using architectures such as convolutional neural networks (CNN) or recurrent neural networks (RNN) to recognize text or numeric information in images. Through this process, the specific content of each field is identified, such as the numeric value of an invoice number or the format of a date. The results of the recognition form a field recognition result set, which contains the precise recognition results of each candidate field. Based on the field recognition result set, the comprehensive matching score of each candidate bill type is calculated. The matching score is calculated based on the field recognition results and the field standards in the template, comparing the difference between the actual recognized content and the expected content in the template. It is expressed using a weighted similarity calculation formula:

[0176] ;

[0177] in, Indicates The comprehensive matching score of the candidate bill types, is the weight of the field, Is Field The matching similarity of is the field standard in the template, is the identified field result. By weighted similarity calculation, the matching score between each candidate bill type and the actual input bill is obtained. Weighted voting is performed on the matching scores of all candidate bill types to select the final bill type. Based on the matching scores of different candidate types, the type with the highest score is selected as the final bill type recognition result. Assume that the bill type matching score vector is , by taking the maximum value, we get the final ticket type:

[0178] ;

[0179] in, is the final bill type recognition result. Indicates taking the bill type with the highest matching score. Combine the bill type recognition result with the field recognition result set to generate a complete bill recognition result. The bill type recognition result provides the classification information of the overall bill, while the field recognition result set provides the content information of the specific field. By combining these two parts of the results, a complete bill parsing result is output, covering all key information such as the bill type, invoice number, amount, date, etc.

[0180] The above describes the bill recognition method based on machine vision in the embodiment of the present invention. The following describes the bill recognition device based on machine vision in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a bill recognition device based on machine vision includes:

[0181] A preprocessing module, used for preprocessing and randomly blocking the input bill image to obtain a preprocessed bill image;

[0182] An extraction module is used to extract multi-scale features from the pre-processed bill image to obtain a multi-scale feature map set;

[0183] An optimization module, which is used to perform progressive attention feature optimization based on a set of multi-scale feature maps to obtain an optimized feature map;

[0184] An analysis module is used to perform semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map;

[0185] The collaborative module is used to perform multi-label collaborative learning based on semantically aligned feature maps to obtain multi-label classification results;

[0186] The recognition module is used to perform feature compression and recognition processing on the multi-label classification results to obtain the bill recognition results.

[0187] Through the synergy of the above components, the introduction of random occlusion preprocessing and multi-scale feature extraction technology improves the model's ability to recognize damaged and deformed bills and enhances the robustness of the system. The progressive attention feature optimization mechanism is adopted to achieve adaptive focusing on the key areas of the bill, and improve the discriminability and representation ability of the features. The semantic alignment local feature analysis method is introduced to effectively integrate text semantic information with visual features, and enhance the model's ability to understand the bill content. Through the multi-label collaborative learning strategy, the correlation between different information fields in the bill is fully utilized to improve the accuracy and consistency of classification. The feature compression and efficient recognition processing technology are adopted to reduce the computational complexity, improve the processing efficiency of the system, and are suitable for large-scale bill recognition applications. The two-stage method combining template matching and fine recognition realizes accurate recognition and field extraction of different types of bills, and has good generalization ability. Through the organic combination of multiple modules, an end-to-end bill recognition process is constructed, which simplifies the deployment and maintenance of the system. The present invention has strong scalability and can adapt to new bill types by updating the model and template library to meet the dynamic needs in practical applications.

[0188] The present invention also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the machine vision-based bill recognition method in the above-mentioned embodiments.

[0189] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the machine vision-based bill recognition method.

[0190] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0192] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bill recognition method based on machine vision, characterized in that: The method comprises: Performing preprocessing and random occlusion processing on the input bill image to obtain a preprocessed bill image; Performing multi-scale feature extraction on the pre-processed bill image to obtain a multi-scale feature map set; Performing progressive attention feature optimization based on the multi-scale feature map set to obtain an optimized feature map; Performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map; Based on the semantic alignment feature map, multi-label collaborative learning is performed to obtain a multi-label classification result; specifically, the method includes: inputting the semantic alignment feature map into a multi-label classifier set to obtain an initial prediction result, wherein the multi-label classifier set includes multiple independent classifiers corresponding to the key information categories of the bill; performing label correlation analysis on the initial prediction result to construct a label correlation matrix, wherein the label correlation matrix represents the dependency relationship between different labels; based on the label correlation matrix, performing conditional random field optimization on the initial prediction result to obtain an optimized prediction result; inputting the optimized prediction result into a label smoothing module, reducing the risk of overfitting through label smoothing technology to obtain a smoothed prediction result, and based on the smoothed prediction result, calculating the confidence score of each label to obtain a label confidence vector; performing dynamic threshold selection on the label confidence vector to determine the optimal decision threshold for each label to obtain a label decision threshold set, and binarizing the smoothed prediction result according to the label decision threshold set to obtain a target classification result; performing label conflict resolution and label completion processing on the target classification result to obtain a multi-label classification result; The multi-label classification result is subjected to feature compression and recognition processing to obtain a bill recognition result; specifically comprising: feature encoding the multi-label classification result to obtain a compressed feature vector, and based on a predefined bill type template library, template matching the compressed feature vector to obtain the most similar bill type candidate set; extracting key field position information for each candidate type in the most similar bill type candidate set to obtain a candidate field position set, and based on the candidate field position set, extracting local features of the input bill image to obtain a local field feature set; inputting the local field feature set into a pre-trained field recognition model, performing fine recognition on each candidate field to obtain a field recognition result set, and based on the field recognition result set, calculating the comprehensive matching score of each candidate bill type to obtain a bill type matching score vector; performing weighted voting on the bill type matching score vector to select the final bill type to obtain a bill type recognition result, and combining the bill type recognition result with the field recognition result set to generate a bill recognition result.

2. The machine vision-based bill recognition method according to claim 1, characterized in that: The step of preprocessing and randomly blocking the input bill image to obtain the preprocessed bill image includes: Performing size standardization processing on the input bill image to obtain a bill image of standard size, and performing color space conversion on the bill image of standard size to obtain a bill image in RGB color space; Based on a preset occlusion ratio range, a random number of rectangular occlusion regions are generated for the bill image in the RGB color space to obtain an occlusion region set; Performing random filling processing on each rectangular area in the occlusion area set to obtain a simulated damaged occlusion area, and superimposing the simulated damaged occlusion area on the bill image in the RGB color space to obtain the bill image after occlusion processing; Performing contrast adjustment on the bill image after the occlusion processing to obtain a bill image with enhanced contrast, and adding random noise to the bill image with enhanced contrast based on a preset noise parameter to obtain a bill image with noise; Edge enhancement is performed on the noisy bill image to obtain a preprocessed bill image.

3. The machine vision-based bill recognition method according to claim 1, characterized in that: The step of performing multi-scale feature extraction on the pre-processed bill image to obtain a multi-scale feature map set includes: Inputting the preprocessed bill image into a deep convolutional neural network for feature extraction to obtain an initial feature map, and based on the initial feature map, obtaining multiple intermediate feature maps of different scales through continuous convolution and pooling operations; Performing feature pyramid network processing on the multiple intermediate feature maps of different scales to obtain a preliminarily fused multi-scale feature map, and performing top-down feature propagation based on the preliminarily fused multi-scale feature map to obtain enhanced high-level semantic features; Performing upsampling processing on the enhanced high-level semantic features to obtain an upsampled feature map having the same size as the low-level feature map, and performing feature fusion on the upsampled feature map with the low-level feature map of the corresponding level to obtain a fused multi-scale feature map; A non-local attention mechanism is performed on the fused multi-scale feature map to obtain a feature map with global context information, and based on the feature map with global context information, feature reconstruction is performed through a multi-layer perceptron to obtain a set of multi-scale feature maps.

4. The machine vision-based bill recognition method according to claim 1, characterized in that: The step of performing progressive attention feature optimization based on the multi-scale feature map set to obtain an optimized feature map includes: Performing grid division on the feature map with the lowest resolution in the multi-scale feature map set to obtain an initial grid feature map; Based on the initial grid feature map, the attention weight of each grid area is calculated by a convolutional neural network to obtain an initial attention score map; Performing softmax normalization processing on the initial attention score map to obtain a normalized attention weight map, and based on the normalized attention weight map, performing weighted processing on the initial grid feature map to obtain a weighted optimized feature map; Upsampling the weighted optimized feature map to the size of the next layer feature map to obtain a target feature map, and performing feature fusion on the target feature map and the original next layer feature map to obtain a fused feature map; Iterative execution is performed based on the fused feature map until feature maps of all scales are processed to obtain a multi-level optimized feature map, and cross-scale feature aggregation is performed on the multi-level optimized feature map to obtain an optimized feature map.

5. The machine vision-based bill recognition method according to claim 4, characterized in that: The method of calculating the attention weight of each grid area through a convolutional neural network based on the initial grid feature map to obtain an initial attention score map includes: Performing channel compression processing on the initial grid feature map to obtain a compressed feature map, and obtaining multi-scale spatial features through a spatial pyramid pooling operation based on the compressed feature map; Performing inter-channel attention calculation on the multi-scale spatial features to obtain channel attention weights, and weighting the multi-scale spatial features based on the channel attention weights to obtain weighted spatial features; Performing a deconvolution operation on the weighted spatial features to obtain an attention feature map having a size consistent with the initial grid feature map, and calculating a local response of each grid area by a sliding window method based on the attention feature map to obtain a local response map; A nonlinear activation function operation is performed on the local response map to obtain an activated attention response map, and based on the activated attention response map, an initial attention score map is obtained through a spatial softmax operation.

6. The machine vision-based bill recognition method according to claim 1, characterized in that: The performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map includes: Performing optical character recognition on the optimized feature graph to extract text information in the bill to obtain a text sequence, and performing semantic encoding on the text sequence based on a pre-trained semantic parsing model to obtain a text semantic vector; Processing the text semantic vector with a self-attention mechanism to obtain an enhanced semantic representation, and constructing a semantic heat map based on the enhanced semantic representation, mapping the semantic information back to the original image space, and obtaining a semantic importance map; Performing Gaussian filtering on the semantic importance map to obtain a smoothed semantic weight map, and weighting the optimized feature map based on the smoothed semantic weight map to obtain a feature map with preliminary semantic alignment; The preliminary semantically aligned feature map is input into a region proposal network to generate candidate regions to obtain a local region feature set, and based on the local region feature set and the enhanced semantic representation, feature fusion is performed through a cross-modal attention mechanism to obtain a semantically aligned feature map.

7. The machine vision-based bill recognition method according to claim 6, characterized in that: The self-attention mechanism is performed on the text semantic vector to obtain an enhanced semantic representation, and based on the enhanced semantic representation, a semantic heat map is constructed to map the semantic information back to the original image space to obtain a semantic importance map, including: Performing multi-head self-attention calculation on the text semantic vector to obtain an attention-weighted semantic vector, and performing feature transformation through a feedforward neural network based on the attention-weighted semantic vector to obtain a preliminary enhanced semantic representation; Performing residual connection and layer normalization on the preliminary enhanced semantic representation to obtain an enhanced semantic representation, and based on the enhanced semantic representation, calculating the relevance score of each semantic unit with the key information category of the bill to obtain a semantic-category relevance matrix; Performing a maximum pooling operation on the semantic-category correlation matrix to extract the most significant category of each semantic unit to obtain a semantic unit category map, and performing a coarse-grained semantic heat map analysis based on the semantic unit category map and position information of the original image to obtain an initial semantic importance map; Performing bilinear interpolation on the initial semantic importance map, upsampling the initial semantic importance map to the same size as the original image, to obtain an upsampled semantic importance map; Adaptive threshold segmentation is performed on the upsampled semantic importance map to obtain a semantic importance map.

8. A bill recognition device based on machine vision, characterized in that: Used to execute the machine vision-based bill recognition method as described in any one of claims 1 to 7, the machine vision-based bill recognition device comprises: A preprocessing module, used for preprocessing and randomly blocking the input bill image to obtain a preprocessed bill image; An extraction module, used for performing multi-scale feature extraction on the pre-processed bill image to obtain a multi-scale feature map set; An optimization module, configured to perform progressive attention feature optimization based on the multi-scale feature map set to obtain an optimized feature map; An analysis module, used for performing semantically aligned local feature analysis on the optimized feature map to obtain a semantically aligned feature map; A collaborative module, used for performing multi-label collaborative learning based on the semantic alignment feature map to obtain a multi-label classification result; The recognition module is used to perform feature compression and recognition processing on the multi-label classification result to obtain a bill recognition result.

Images