Bill scene recognition method, device, equipment and storage medium

Abstract

The present application relates to the field of image processing technology, and discloses a bill scene recognition method, device, equipment and storage medium, the method comprising: processing the bill image to obtain initial feature data; performing local affine transformation and illumination simulation to obtain an enhanced data set; performing multi-layer feature extraction to obtain multi-scale fusion features; constructing a graph structure of the bill table and performing node feature aggregation processing to obtain table structure data; performing spatial transformation and bidirectional sequence encoding, and decoding processing to obtain bill text data; performing integration, entity linking and information completion to obtain structured bill recognition results, thereby being able to effectively process the multi-scale features of the bill image, cope with various complex table structures, and accurately recognize diverse text content. In addition, considering the particularity of bill processing, the method can also use domain knowledge to complete and verify information, ensuring the accuracy and completeness of the recognition results.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a bill scene recognition method, device, equipment and storage medium. Background Art

[0002] Bill scene recognition is of great significance to improving bill processing efficiency and accuracy. Traditional manual processing methods can no longer meet the needs of modern enterprises. However, bill scene recognition faces many challenges, such as diverse bill types, complex formats, and severe background interference, which greatly increase the difficulty of recognition.

[0003] Existing bill recognition technology still has some limitations when dealing with complex scenarios. For example, for bills with complex table structures, traditional OCR technology has difficulty accurately recognizing and extracting structured information; for bills with diverse fonts and poor printing quality, the recognition accuracy of existing methods is often poor. In addition, common problems in bills, such as tilt, deformation, and uneven lighting, also bring great challenges to recognition. Summary of the invention

[0004] The present application provides a bill scene recognition method, device, equipment and storage medium, which can effectively process the multi-scale features of bill images, cope with various complex table structures, and accurately recognize diverse text contents. In addition, considering the particularity of bill processing, the method can also use domain knowledge to complete and verify information, ensuring the accuracy and completeness of the recognition results.

[0005] The first aspect of the present application provides a bill scene recognition method, the bill scene recognition method comprising:

[0006] Perform multi-scale convolution processing on the bill image to obtain initial feature data;

[0007] Performing local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set;

[0008] Performing multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features;

[0009] Constructing a graph structure of a bill table based on the multi-scale fusion features, and performing node feature aggregation processing through iterative graph convolution to obtain table structure data;

[0010] Performing spatial transformation and bidirectional sequence encoding on the multi-scale fusion features, and decoding processing in combination with a dynamic attention mechanism to obtain bill text data;

[0011] Based on the bill domain knowledge base, the table structure data and the bill text data are integrated, entity linked and information completed to obtain a structured bill recognition result.

[0012] A second aspect of the present application provides a bill scene recognition device, the bill scene recognition device comprising:

[0013] A processing module, used for performing multi-scale convolution processing on the bill image to obtain initial feature data;

[0014] A transformation module, used for performing local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set;

[0015] An extraction module, used for performing multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features;

[0016] A construction module is used to construct a graph structure of the bill table based on the multi-scale fusion features, and perform node feature aggregation processing through iterative graph convolution to obtain table structure data;

[0017] A decoding module, used to perform spatial transformation and bidirectional sequence encoding on the multi-scale fusion features, and perform decoding processing in combination with a dynamic attention mechanism to obtain bill text data;

[0018] The integration module is used to integrate, entity link and complete the information of the table structure data and the bill text data based on the bill field knowledge base to obtain a structured bill recognition result.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned bill scene recognition method.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned bill scene recognition method.

[0021] Compared with the prior art, the present application has the following beneficial effects: through multi-scale convolution processing and data enhancement technology, the adaptability of the model to various complex bill scenarios is improved, and the robustness of recognition is enhanced. The use of multi-layer feature extraction and fusion processing can effectively capture the different levels of features of the bill image and improve the recognition ability of complex structures and detailed information. The introduction of graph structure and iterative graph convolution technology significantly improves the understanding and information extraction capabilities of complex table structures. Combining spatial transformation and dynamic attention mechanism, the model's recognition ability for tilted and deformed text is enhanced, and the accuracy of text recognition is improved. By integrating the bill field knowledge base, entity linking and information completion are achieved, which greatly improves the completeness and accuracy of the recognition results. This method has strong versatility and can be applied to various types of bill recognition tasks, reducing the need to develop specialized algorithms for different bill types. It improves the efficiency of the overall recognition process and is suitable for applications in large-scale bill processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0024] Figure 1 It is a flowchart of a bill scene recognition method provided by an embodiment of the present invention;

[0025] Figure 2 It is a schematic block diagram of the structure of the bill scene recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 , an embodiment of the bill scene recognition method in the embodiment of the present application includes:

[0030] Step 100: Perform multi-scale convolution processing on the bill image to obtain initial feature data;

[0031] It is understandable that the execution subject of the present application may be a bill scene recognition device, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0032] Specifically, the input bill image is preprocessed, including resizing, denoising and brightness normalization, to obtain standardized image data. The standardized image data is input into the first convolution layer, and the image is convolved with a 7×7 convolution kernel to generate the first convolution feature map. In order to reduce the amount of data and retain the significant features of the image, the first convolution feature map is subjected to a maximum pooling operation, which can effectively reduce the dimension of the feature map while keeping the key features from being lost, thereby obtaining a downsampled feature map. The downsampled feature map is input into the second convolution layer, and a 5×5 convolution kernel is used to generate a second convolution feature map to capture the mid-scale features in the bill image. The second convolution feature map is subjected to an average pooling operation. Average pooling helps to retain more global information while reducing the image dimension, thereby obtaining a compressed feature map. The compressed feature map is input into the third convolution layer, and a 3×3 convolution kernel is used to perform a convolution operation to extract fine-grained features in the bill image, thereby generating a third convolution feature map. In order to improve the ability to capture information of different scales, spatial pyramid pooling is performed on the third convolution feature map. Spatial pyramid pooling is a technology that can process image features at multiple scales. It captures image details and global features at different resolutions by partitioning and pooling feature maps, generating multi-scale feature vectors. The multi-scale feature vectors are input into the fully connected layer for processing to obtain a global feature representation. Batch normalization is performed on the feature representation so that these features maintain a stable and consistent distribution in different batches, improving the convergence speed and stability of the model. Nonlinear activation functions are performed on the normalized features. Common activation functions include ReLU, etc. Nonlinear factors are introduced to enable the model to better fit complex image features and ultimately obtain initial feature data.

[0033] Step 200: Perform local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set;

[0034] Specifically, random area selection is performed on the initial feature data. By randomly selecting specific areas, the diversity of transformation and enhancement is ensured, so that the model can better adapt to various changes in the bill scene. After the area to be transformed is selected, the affine transformation matrix analysis is performed on the area based on the preset transformation parameter range. The local transformation matrix is ​​calculated. The transformation matrix determines how the selected area is affine transformed, such as rotation, scaling, translation, etc., to obtain the local transformation matrix. According to the local transformation matrix, the area to be transformed is affine transformed, and the morphology of the area will change locally, and the local deformation feature is obtained. In order to fuse the local change with the overall feature data, a specific fusion method is used to combine the local deformation feature with the initial feature data to form a global deformation feature. Without destroying the original information, the local change information of the image is enhanced, so that the model can still maintain the recognition accuracy when facing deformed or distorted bill images. Based on the predefined illumination model, the illumination change parameter of the global deformation feature is calculated to obtain the illumination adjustment factor, which is used to adjust the brightness of the image. The purpose of illumination simulation is to enhance the adaptability of the model to bill images under different illumination conditions. The brightness of the global deformation features is adjusted by the illumination adjustment factor to generate brightness adjustment features, so that the overall illumination effect of the image is more diverse. Based on the Gaussian distribution, the brightness adjustment features are masked for local highlights and shadows. By generating an illumination effect mask, the illumination unevenness in different regions, such as the highlight part and the shadow part, is simulated. According to the illumination effect mask, the brightness adjustment features are locally enhanced to obtain simulated illumination features, which simulate the local illumination differences that may occur in bills under different light sources. In order to improve the image enhancement effect, the simulated illumination features are contrast adjusted. By adjusting the contrast, the details of the image can be made more prominent while maintaining the overall visual consistency, and contrast enhancement features are generated. Based on the contrast enhancement features, multiple enhanced samples of different degrees are generated. Each enhanced sample represents a combination of different illumination, contrast or affine transformation, resulting in a diverse enhanced data set.

[0035] Step 300: Perform multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features;

[0036] It should be noted that the enhanced data set is input into the shallow convolutional network for processing, and the basic features in the image are extracted to obtain low-level features, including relatively simple visual information such as the edge and texture of the image, and obtain a low-level feature map. In order to enhance the expression ability of low-level features, spatial pyramid pooling is performed based on the low-level feature map. Spatial pyramid pooling is a technology that performs pooling operations at different scales, which can capture multi-scale information while ensuring feature resolution and generate multi-scale low-level features. The low-level feature map is input into the middle-layer convolutional network. In the middle-layer convolutional network, some intermediate features such as shape and structure can be captured. After being processed by the middle-layer convolutional network, an intermediate feature map is obtained. In order to simplify the expression of features and ensure their dimensional consistency, adaptive average pooling is performed based on the intermediate feature map. Adaptive average pooling adjusts feature maps of different sizes to a fixed dimension, so that in the subsequent processing process, the performance of the model will not be affected by inconsistent feature dimensions. Through this operation, intermediate features of fixed dimensions are obtained. The intermediate feature map is input into the deep convolutional network for processing to extract high-level features in the image and capture more abstract concepts and patterns, such as the overall shape of an object or complex contextual information. After being processed by the deep convolutional network, a high-level feature map is obtained. In order to refine the high-level features, a global maximum pooling operation is performed based on the high-level feature map to extract the most significant features from the entire image. These features usually represent the key information in the image and generate global high-level features. Channel attention calculations are performed on multi-scale low-level features, fixed-dimensional intermediate features, and global high-level features. The channel attention mechanism can assign different weights to channels of different features, emphasize key channel information, weaken irrelevant channels, and enable the model to capture important features more accurately. Through this calculation, weighted features are obtained. Feature interaction operations are performed based on weighted features. Associations are established between features at different levels so that low-level features, intermediate features, and high-level features can complement each other to form a comprehensive feature expression and obtain interactive features. Nonlinear transformation and normalization are performed on interactive features. Nonlinear transformation usually uses activation functions such as ReLU to introduce the nonlinear ability of the model and enhance the expressiveness of the features; normalization ensures that the features maintain a consistent distribution in different batches, improving the stability and convergence speed of the model. After this process, the standardized fusion features are obtained. Residual connection and feature recalibration operations are performed based on the standardized fusion features. Residual connection can effectively avoid the gradient vanishing problem and ensure that deep features are well preserved and transmitted during back propagation; feature recalibration further optimizes the features so that the final features are more in line with the recognition requirements of the model, and finally generate multi-scale fusion features.

[0037] Step 400: construct a graph structure of the bill table based on multi-scale fusion features, and perform node feature aggregation processing through iterative graph convolution to obtain table structure data;

[0038] Specifically, edge detection is performed on the multi-scale fusion features to identify the table lines in the bill image and generate a table line feature map. Hough transform is performed based on the table line feature map. Hough transform is a commonly used image processing technology that can convert lines in an image into geometric representations through a line detection algorithm to generate candidate regions for table cells. Non-maximum suppression is performed on the generated candidate regions for table cells to eliminate redundant information and retain the most significant table cell boundaries. Non-maximum suppression can effectively remove redundant overlapping areas and obtain accurate table cell locations. Based on the accurate table cell locations, an initial graph structure is constructed to generate a topological graph of the table. Each node in the topological graph represents a table cell, and the connection between nodes represents the adjacency relationship between cells, forming an initial structural representation of the table. Each node in the topological graph is input into the regional feature extraction network for feature extraction, and the local features of each table cell are extracted to form the initial features of the node. In order to strengthen the relationship representation between nodes, the similarity between each two nodes is calculated based on the initial features of the nodes, and the adjacency matrix of the graph is generated to reflect the connection strength between the nodes in the graph. The adjacency matrix is ​​analyzed by the graph attention mechanism. Through the graph attention mechanism, each connection in the adjacency matrix can be weighted to emphasize the important associations between nodes and weaken irrelevant connections. A weighted adjacency matrix is ​​generated, which can more accurately express the relationship between cells in the table. Based on the weighted adjacency matrix and the initial features of the nodes, a graph convolution operation is performed. Graph convolution is a technology that can aggregate node features in a graph structure. By convolving the information of neighboring nodes, the feature expression of the node is updated so that the features of each node not only contain its own information, but also integrate the features of surrounding nodes. Through multiple iterations of graph convolution operations, each round of iteration aggregates the information between nodes, making the feature expression of the node more complete and accurate. After multiple iterations, the final node representation is obtained. Based on the final node representation, graph structure decoding is performed. The graph structure is restored to a table structure data containing cell type and content, and the type of each cell (such as title, data, etc.) and the text or other information contained therein are accurately identified.

[0039] Step 500: Perform spatial transformation and bidirectional sequence encoding on the multi-scale fusion features, and perform decoding processing in combination with a dynamic attention mechanism to obtain bill text data;

[0040] Specifically, the multi-scale fusion features are input into the spatial transformation network, and the input feature map is corrected to correct the geometric distortion caused by shooting angle, deformation or tilt in the bill image, so as to obtain the corrected feature map. Based on the corrected feature map, adaptive binarization is performed to separate the text area and non-text area in the image, and a binary text area mask is generated to clearly indicate which parts are text content. Adaptive binarization can dynamically adjust the threshold according to different lighting conditions to ensure accurate segmentation of the text area. Connected domain analysis is performed on the generated text area mask, and each candidate text line area is identified by analyzing the shape, size and position of the connected area. Based on the candidate text line area, the corrected feature map is cropped to extract the feature sequence corresponding to each text line, which contains the local details and overall structural information of each line of text. The text line feature sequence is input into the bidirectional long short-term memory (LSTM) network. Bidirectional LSTM is a sequence model that can simultaneously capture the forward and backward context information in the text line. The obtained context-aware feature representation contains the features of each character itself and also integrates the context information of the adjacent characters before and after the character. Based on the context-aware feature representation, the attention weights are calculated to generate the initial attention distribution. The role of the attention mechanism is to assign different weights to each position in the sequence so that the model can focus on the most relevant part at the moment. In order to optimize the attention distribution, a temperature scaling operation is performed on the initial attention distribution. By adjusting the smoothness of the attention distribution, the flexibility of the model in processing different characters is enhanced. After the temperature scaling process, the adjusted attention distribution is obtained. Based on the adjusted attention distribution and the context-aware feature representation, a weighted sum operation is performed to re-weight the importance of different characters to generate a comprehensive context vector. The context vector is input into the multi-layer perceptron network for processing. The multi-layer perceptron network can classify the context vector and output the probability distribution of the character category. Each character in the bill text is identified and the probability of each character belonging to the category is given. Based on the probability distribution of the character category, beam search decoding is performed. Beam search is a search algorithm that can select the best path among multiple decoding paths to generate the final bill text data.

[0041] Step 600: Based on the bill domain knowledge base, the table structure data and the bill text data are integrated, entity linked and information completed to obtain a structured bill recognition result.

[0042] Specifically, the table structure data and the bill text data are merged to generate initial fused data. By combining the table structure and text content, data from different sources are unified into one format. Field mapping is performed on the initial fused data based on the predefined bill template. The bill template defines the standard field structure in the bill, such as date, amount, payee, etc. Through field mapping, the initial fused data is converted into structured bill data to ensure that each data field is accurately located and classified. Named entity recognition is performed on the entity information in the structured bill data. Named entity recognition is a natural language processing technology used to identify specific categories of entities from text, such as company name, address or amount. Through this process, a list of candidate entities is generated. Entity disambiguation is performed based on the bill domain knowledge base. Entity disambiguation accurately maps polysemous entities to unique entity identifiers through context information. For example, if multiple company names are mentioned in the same bill, disambiguation is used to determine the specific company referred to by each company name. After disambiguation, a unique entity identifier is obtained. Based on the unique entity identifier, attribute information related to the entity, such as company address, legal person information, etc., is retrieved from the bill domain knowledge base. Enhanced bill data is generated by fusing entity attribute data with structured bill data. Missing value detection is performed on enhanced bill data based on preset bill field integrity rules. Through detection, the fields to be completed are identified, and a list of fields to be completed is generated. Based on the list, candidate completion information related to these fields is retrieved from the bill domain knowledge base. The completion information in the knowledge base may include data in historical bills or general information related to the identified entities. In order to ensure the accuracy of the completion information, the candidate completion data is scored for contextual relevance. According to the specific context of the current bill, the degree of fit of each candidate completion data with it is calculated. Finally, the completion information with the highest relevance score is selected as the optimal completion information. Based on the optimal completion information, the missing fields in the enhanced bill data are completed to obtain a complete structured bill recognition result.

[0043] In the embodiments of the present application, multi-scale convolution processing and data enhancement technology are used to improve the adaptability of the model to various complex bill scenarios and enhance the robustness of recognition. The use of multi-layer feature extraction and fusion processing can effectively capture the different levels of features of the bill image and enhance the recognition ability of complex structures and detailed information. The introduction of graph structure and iterative graph convolution technology significantly improves the understanding and information extraction capabilities of complex table structures. Combining spatial transformation and dynamic attention mechanism, the model's recognition ability for tilted and deformed text is enhanced, and the accuracy of text recognition is improved. By integrating the bill field knowledge base, entity linking and information completion are achieved, which greatly improves the completeness and accuracy of the recognition results. This method has strong versatility and can be applied to various types of bill recognition tasks, reducing the need to develop specialized algorithms for different bill types. It improves the efficiency of the overall recognition process and is suitable for applications in large-scale bill processing scenarios.

[0044] In a specific embodiment, the process of executing step 100 may specifically include the following steps:

[0045] Preprocess the input bill image to obtain standardized image data;

[0046] The standardized image data is input into the first convolution layer, a 7×7 convolution kernel is used for convolution operation to obtain the first convolution feature map, and a maximum pooling operation is performed on the first convolution feature map to obtain a downsampled feature map;

[0047] The downsampled feature map is input into the second convolution layer, a 5×5 convolution kernel is used for convolution operation to obtain a second convolution feature map, and an average pooling operation is performed on the second convolution feature map to obtain a compressed feature map;

[0048] The compressed feature map is input into the third convolution layer, a 3×3 convolution kernel is used to perform convolution operation to obtain the third convolution feature map, and spatial pyramid pooling is performed on the third convolution feature map to obtain a multi-scale feature vector;

[0049] The multi-scale feature vector is input into the fully connected layer for processing to obtain feature representation, and batch normalization operation is performed on the feature representation to obtain normalized features;

[0050] Perform nonlinear activation function operation on the normalized features to obtain the initial feature data.

[0051] Specifically, the input bill image is preprocessed, including grayscale, resizing, denoising and other operations. Through these processes, the effects of noise, non-uniform lighting and other factors in the image are eliminated to obtain standardized image data. The purpose of standardized images is to convert images of different sizes and formats into input data of the same scale, so that subsequent convolution operations can be performed on a consistent scale. The standardized image data is input to the first convolution layer. In this layer, a 7×7 convolution kernel is used. The function of the convolution kernel is to scan the entire image and extract local features in different areas of the image through a weight sharing mechanism. Through the formula ,in, represents the input normalized image data, is a 7×7 convolution kernel, is the bias term, is the first convolution feature map after the convolution operation. By sliding across the entire image The convolution kernel of size 5x5 can effectively capture the local patterns and edge information in the image. Perform the maximum pooling operation on the first convolution feature map. By selecting the maximum value in the pooling area, the spatial dimension of the feature map is reduced while retaining the most significant features. After maximum pooling, the downsampled feature map is obtained, which makes the size of the feature map smaller, reduces the computational complexity, and enhances the model's focus on key features. The downsampled feature map is input to the second convolution layer using a convolution kernel of size 5x5. Smaller convolution kernels can capture local patterns in the image in more detail, especially important features such as text edges and table lines in the bill image. Similar to the operation of the previous layer, the second convolution layer is implemented through the formula ,in is the second convolutional feature map, is a 5×5 convolution kernel, Is the bias term. The second convolution feature map obtained after convolution contains more fine-grained image features. In order to compress the image information, an average pooling operation is performed on the second convolution feature map. Average pooling reduces the spatial dimension by taking the average value in the pooling area. This operation helps to smooth the feature map and retain global information while reducing excessive detail interference, resulting in a compressed feature map with a smaller dimension and more compact information. The compressed feature map is input into the third convolution layer. A 3×3 convolution kernel is used for convolution operations. 3×3 convolution kernels are often used to extract finer local features, especially small-scale patterns in images, such as important information such as characters and numbers in bills. The third convolution feature map after convolution is obtained by the formula Generate, where is a 3×3 convolution kernel, is the third convolution feature map. In order to capture feature information at different scales, spatial pyramid pooling is performed on the third convolution feature map. The feature map is divided into sub-regions of different scales, and pooling operations are performed separately to generate feature vectors of multiple scales. These feature vectors of different scales can capture global and local information in the image, effectively improving the feature expression ability of the model. After spatial pyramid pooling, a multi-scale feature vector is obtained, which contains image information from different levels and resolutions. The multi-scale feature vector is input into the fully connected layer for processing, and the local features from the convolution layer are integrated to generate a global feature representation. Through matrix operations ,in is the weight matrix of the fully connected layer, is a multi-scale feature vector, is the bias term, and the feature representation is obtained. Batch normalization is performed on the feature representation to standardize the features so that the features maintain a consistent distribution in different batches, avoiding the problem of gradient vanishing or gradient exploding, and improving the training efficiency and stability of the model. After batch normalization, the normalized features are obtained. A nonlinear activation function is performed on the normalized features. The RelU function is commonly used as a nonlinear activation function, and its formula is , the part of the feature that is less than 0 is set to 0, and the positive value is retained, thereby introducing nonlinear factors and enabling the model to better fit complex patterns. After being processed by the activation function, the initial feature data is obtained.

[0052] In a specific embodiment, the process of executing step 200 may specifically include the following steps:

[0053] Randomly select an area of ​​the initial feature data to obtain an area to be transformed, and perform affine transformation matrix analysis on the area to be transformed based on a preset transformation parameter range to obtain a local transformation matrix;

[0054] Perform affine transformation on the area to be transformed according to the local transformation matrix to obtain local deformation features, and fuse the local deformation features with the initial feature data to obtain global deformation features;

[0055] Based on the predefined illumination model, the illumination change parameter of the global deformation feature is calculated to obtain the illumination adjustment factor, and the brightness of the global deformation feature is adjusted according to the illumination adjustment factor to obtain the brightness adjustment feature;

[0056] Based on Gaussian distribution, the brightness adjustment feature is processed with local highlight and shadow masks to obtain the lighting effect mask;

[0057] Performing local illumination enhancement on the brightness adjustment feature according to the illumination effect mask to obtain a simulated illumination feature, and performing contrast adjustment on the simulated illumination feature to obtain a contrast enhancement feature;

[0058] Based on the contrast enhancement feature, multiple enhanced samples with different degrees are generated to obtain an enhanced data set.

[0059] Specifically, a random region selection is performed on the initial feature data, and a specific region is selected from the entire image for subsequent processing. The randomness ensures that the model can handle a variety of different image change scenarios. By randomly selecting regions in the image, the transformation and enhancement effects are more diverse and the generalization ability of the model is improved. The selected region is the region to be transformed. Based on the preset affine transformation parameter range, the affine transformation matrix analysis is performed on the region to be transformed. Affine transformation is a common geometric transformation operation that can change the shape and position of an image by rotation, scaling, translation, etc. The affine transformation matrix is ​​usually expressed as:

[0060] ;

[0061] in, , , and Determines the rotation and scaling of the image, and Determines the translation of the image. For each area to be transformed, these parameters are randomly selected according to the set transformation parameter range to generate a local transformation matrix . Through this matrix, the image is rotated, scaled or distorted locally to simulate the changes that may occur in the image in various practical application scenarios. According to the generated local transformation matrix, the area to be transformed is affine transformed to obtain local deformation features. The local deformation features are fused with the original initial feature data to generate global deformation features. The fusion operation can be achieved through simple superposition, weighted summation or other more complex fusion mechanisms. The purpose of this process is to combine local changes with global features, introduce local changes while maintaining the overall structure of the image, so that the model can recognize various changes in the image. Simulate illumination changes based on a predefined illumination model. Through the set illumination model, the illumination change parameters of each pixel are calculated to generate an illumination adjustment factor. The illumination adjustment factor represents the change in illumination intensity, which is usually calculated based on multiple factors such as ambient lighting, shadows, highlights, etc. The specific formula can be expressed as:

[0062] ;

[0063] in, is the brightness value after light adjustment, is the original brightness value, is a function that describes the change in illumination, based on the pixel position of the image Generate different adjustment factors. Through this lighting adjustment operation, generate brightness adjustment features to simulate the brightness changes of images under different light sources or lighting conditions. Perform mask processing of local highlights and shadows on the brightness adjustment features based on Gaussian distribution. Gaussian distribution is used to simulate local lighting changes under natural lighting conditions and generate a lighting effect mask. This mask can be expressed as:

[0064] ;

[0065] in, is the value of the lighting mask, and Indicates the coordinates of the highlight or shadow center. Indicates the extent of the illumination range expansion. In this way, a distribution map of local highlights or shadows is generated to represent the local changes in illumination in the image. According to the generated illumination effect mask, the brightness adjustment feature is locally enhanced to simulate the changes in brightness and darkness under different light source intensities, so that the image is closer to the complex illumination conditions in reality. After local illumination enhancement, a simulated illumination feature is generated to improve the illumination diversity of the image. In order to enhance the visual effect of the image, the simulated illumination feature is contrast adjusted. Contrast adjustment can make the details of the image clearer by adjusting the difference between the bright and dark areas of the image. In this way, a contrast enhancement feature is generated, and multiple enhanced samples of different degrees are generated based on the contrast enhancement feature. By performing multiple random combinations of the above-mentioned affine transformation, illumination adjustment and contrast enhancement, multiple different enhanced samples are generated, which can simulate different scene changes in the real world, such as illumination changes, perspective changes, image distortion, etc. Enhanced data samples help train a more robust deep learning model, so that it can show higher recognition accuracy when facing complex image scenes.

[0066] In a specific embodiment, the process of executing step 300 may specifically include the following steps:

[0067] The enhanced dataset is input into a shallow convolutional network for processing to obtain a low-level feature map, and spatial pyramid pooling is performed based on the low-level feature map to obtain multi-scale low-level features;

[0068] The low-level feature map is input into the middle-level convolutional network for processing to obtain the middle-level feature map, and adaptive average pooling is performed based on the middle-level feature map to obtain the fixed-dimensional middle-level features;

[0069] The intermediate feature map is input into the deep convolutional network for processing to obtain the high-level feature map, and the global maximum pooling is performed based on the high-level feature map to obtain the global high-level features;

[0070] Perform channel attention calculation on multi-scale low-level features, fixed-dimensional mid-level features, and global high-level features to obtain weighted features, and perform feature interaction operations based on the weighted features to obtain interactive features;

[0071] The interaction features are nonlinearly transformed and normalized to obtain standardized fusion features, and residual connection and feature recalibration are performed based on the standardized fusion features to obtain multi-scale fusion features.

[0072] Specifically, the enhanced dataset is input into a shallow convolutional network to extract low-level features in the image, such as edges, textures, etc. The convolution kernel of this layer is usually small, such as 3×3 or 5×5, to capture detailed information in the image. ,in represents the input augmented dataset, is a shallow convolution kernel, is the bias term, Represents the low-level feature map after the convolution operation. The low-level feature map mainly contains local information of the image, such as character contours or table lines in bills. Spatial pyramid pooling is performed based on the low-level feature map, and image features are pooled at different scales to obtain multi-scale low-level features. Spatial pyramid pooling divides the image into different regions and performs pooling in these regions respectively. For example, the feature map is divided into 1×1, 2×2, and 4×4 grids, and pooling operations are performed in each grid to generate feature vectors of different scales. The low-level feature map is input into the middle-layer convolutional network for processing. The convolution kernel of the middle-layer convolutional network is usually slightly larger, such as 5×5 or 7×7, which is used to extract mid-level features in the image. Mid-level features are usually manifested as more complex patterns, such as character combinations or more complex geometric shapes in bills. Through the formula ,in represents the middle-layer convolution kernel, It is a mid-level feature map. The convolution operation of this layer enables the model to gradually transition from low-level edges and textures to capturing higher-level image structure information. In order to ensure that the mid-level features have a fixed dimension, adaptive average pooling is performed on the mid-level feature map. The main purpose of adaptive average pooling is to compress feature maps of different sizes to the same fixed dimension while maintaining the integrity of the feature map. The pooling operation generates mid-level features of fixed dimension. The mid-level feature map is input into a deep convolutional network to extract high-level features. The deep convolutional network can capture global patterns in the image, such as the layout of the entire bill, the table structure, or complex character interactions. Through the formula Calculate high-level feature maps. Deep convolutional networks can capture more abstract features that represent the global structure of the bill. Perform global maximum pooling operations based on high-level feature maps to extract the most significant features from the entire feature map, which can effectively reduce the dimension of the feature map and retain important global information. The formula for global maximum pooling is:

[0073] ;

[0074] in, represents the pooling operation, Represents the high-level feature map after deep convolution. Global maximum pooling can generate global high-level features to represent the overall layout of the bill or the most prominent pattern. Channel attention calculation is performed on multi-scale low-level features, fixed-dimensional mid-level features, and global high-level features, and different weights are assigned to the channels of each feature map, so that the model can focus on more useful information. By calculating the importance of each channel, weighted features are generated. The formula is:

[0075] ;

[0076] in, is the attention weight of the channel, is the channel weight matrix, is a feature channel. Through weighted operations, the model can highlight those feature channels that are more important in the recognition task and weaken irrelevant information. Feature interaction operations are performed based on weighted features. The features from different convolutional layers are fused so that the model can make full use of the diversity and complementarity of features in each layer. Low-level features, mid-level features, and high-level features capture the local information, pattern structure, and global layout of the image respectively. Through feature interaction operations, these features at different levels can complement and enhance each other. For example, low-level edge features can provide support for mid-level and high-level pattern recognition, while high-level features can provide a more global context for low-level features. The formula for feature interaction can be expressed as:

[0077] ;

[0078] in, is an interactive feature, , and Represent multi-scale low-level features, fixed-dimensional mid-level features, and global high-level features, respectively, and , and is the corresponding weight matrix, which is used to adjust the weight of each layer of features during the fusion process. After the feature interaction, nonlinear transformation and normalization are performed. Nonlinear transformation usually uses an activation function such as ReLU, and its formula is:

[0079] ;

[0080] The role of the activation function is to introduce nonlinear factors so that the model can better fit complex patterns. After the nonlinear transformation, the features are normalized, usually using batch normalization. The role of normalization is to adjust the distribution of feature values ​​to a standard normal distribution with a mean of 0 and a variance of 1, thereby improving the training efficiency and stability of the model and obtaining standardized fusion features. Residual connection and feature recalibration operations are performed based on the standardized fusion features. Residual connection is an effective network design strategy that can prevent the gradient vanishing problem and maintain information transfer in deep networks. The formula for residual connection is:

[0081] ;

[0082] in, is the residual feature, is the input feature. The residual connection adds the input feature to the interaction feature to ensure that the original information is not lost during the feature fusion process. The feature recalibration operation adjusts the features to strengthen the most important features. The purpose of recalibration is to ensure that the final output features are more sensitive to the target task and can better meet the needs of actual application scenarios. Through the above steps, multi-scale fusion features are finally obtained.

[0083] In a specific embodiment, the process of executing step 400 may specifically include the following steps:

[0084] Perform edge detection on the multi-scale fusion features to obtain a table line feature map, and perform Hough transform based on the table line feature map to obtain the table cell candidate area;

[0085] Perform non-maximum suppression processing on the table cell candidate area to obtain the precise table cell position, and construct the initial graph structure based on the precise table cell position to obtain the table topology map;

[0086] Input each node in the table topology graph into the regional feature extraction network for feature extraction to obtain the initial features of the node;

[0087] The similarity matrix between nodes is calculated based on the initial features of the nodes to obtain the adjacency matrix of the graph, and the graph attention mechanism is analyzed on the adjacency matrix of the graph to obtain the weighted adjacency matrix;

[0088] Perform graph convolution operations based on the weighted adjacency matrix and the initial node features to obtain updated node features, and perform multiple iterative graph convolution operations on the updated node features to obtain the final node representation;

[0089] Graph structure decoding is performed based on the final node representation to obtain table structure data including cell type and content.

[0090] Specifically, edge detection is performed on the multi-scale fusion features to obtain a table line feature map. Edge detection clearly outlines the outline of the table by identifying significant edges in the image, especially the line structure in the bill or table. Commonly used edge detection algorithms such as the Canny algorithm can effectively process noise in the image and accurately extract horizontal and vertical lines in the table. Hough transform is performed based on the table line feature map. Hough transform is a commonly used image processing technology suitable for line detection. It finds the straight lines in the image by mapping the points in the image to the straight line representation in the Hough space. Specifically, the formula of Hough transform is:

[0091] ;

[0092] in, represents the vertical distance from the line to the origin of the image, represents the angle of the straight line, and is the pixel coordinate in the image. By parameterizing the line in Hough space and By performing statistics, all straight lines in the image can be detected. Based on the detected straight lines, the Hough transform can generate candidate regions for table cells, which are possible cell boundaries in the table. Non-maximum suppression is performed on the candidate regions to eliminate overlapping or redundant boundaries, retain only the most prominent boundaries, and obtain more accurate table cell locations. Non-maximum suppression generates accurate table cell locations by comparing the feature values ​​of adjacent regions, retaining local maxima, and removing those insignificant candidate boundaries. Based on the accurate table cell locations, the initial graph structure of the table is constructed. The initial graph structure is usually represented as a topological graph, in which each node corresponds to a table cell, and the edges between nodes represent the connection relationship between cells. The constructed table topology clearly reflects the spatial layout and connection mode of each cell in the table. Each node in the table topology graph is input into the regional feature extraction network for feature extraction. The regional feature extraction network performs feature analysis on each cell region and extracts the initial feature vector of each cell. The initial feature vector contains the geometric information of the cell, as well as the text, symbols, or numbers that may be contained in the cell. Through the processing of the feature extraction network, each node obtains the corresponding initial feature. In order to analyze the relationship between nodes, the similarity matrix between nodes is calculated based on the initial features of the nodes. The similarity matrix is ​​used to represent the similarity between every two nodes. Commonly used similarity calculation methods include Euclidean distance or cosine similarity. The formula is:

[0093] ;

[0094] in, and Respectively represent nodes and The characteristic vector of Is a node and The matrix reflects the relationship between cells in the table, helping the model to better understand the structure of the table. Based on the similarity matrix, the graph attention mechanism is analyzed for the adjacency matrix of the graph. The graph attention mechanism can highlight the node connections that play a key role in the graph structure by assigning different weights to each node. Through this mechanism, a weighted adjacency matrix is ​​generated. , the formula is:

[0095] ;

[0096] in, is a learnable weight matrix, is the original similarity matrix, is a weighted adjacency matrix. The weighted adjacency matrix can strengthen the important node connections in the graph and weaken those unimportant or irrelevant connections. The graph convolution operation is performed based on the weighted adjacency matrix and the initial features of the nodes. The graph convolution operation is an operation that can aggregate node features in the graph structure and update the feature vector of each node by combining the information of neighboring nodes. The formula for graph convolution is:

[0097] ;

[0098] in, Indicates The node feature matrix of the layer, is the weighted adjacency matrix, It is The weight matrix of the layer, is a nonlinear activation function. Through this operation, the features of each node not only contain its own information, but also incorporate the information of its neighboring nodes, making the feature representation richer. In order to improve the feature expression ability of the model, the updated node features are subjected to multiple iterations of graph convolution processing. Each iteration will gradually aggregate the information of neighboring nodes into the target node, so that the node features gradually contain more global information. After multiple iterations, the node feature representation containing rich contextual information is finally obtained. Based on the final node features, graph structure decoding is performed. According to the features and connection relationships of the nodes, the specific structure of the table is reconstructed. Each node not only represents a table cell, but also contains the type and content information of the cell. Through the decoding operation, table structure data containing cell type (such as title, data, comments, etc.) and content is generated.

[0099] In a specific embodiment, the process of executing step 500 may specifically include the following steps:

[0100] Perform spatial transformation network on the multi-scale fusion features to obtain the corrected feature map, and perform adaptive binarization processing based on the corrected feature map to obtain the text area mask;

[0101] Perform connected domain analysis on the text region mask to obtain candidate text line regions, and crop the corrected feature map based on the candidate text line regions to obtain a text line feature sequence;

[0102] Input the text line feature sequence into the bidirectional LSTM network to extract context features, obtain context-aware feature representation, and calculate the attention weight based on the context-aware feature representation to obtain the initial attention distribution;

[0103] Perform a temperature scaling operation on the initial attention distribution to obtain an adjusted attention distribution, and perform a weighted sum based on the adjusted attention distribution and the context-aware feature representation to obtain a context vector;

[0104] The context vector is input into the multi-layer perceptron network for processing to obtain the character category probability distribution, and beam search decoding is performed based on the character category probability distribution to obtain the bill text data.

[0105] Specifically, the input multi-scale fusion features are processed. The spatial transformation network is a key component for correcting geometric distortion of images. It performs affine transformation on specific areas of the input image to correct irregularities in the image. Through this network, a corrected feature map is obtained. The corrected feature map can better express the text and table content in the bill image and reduce the deviation caused by shooting angle, image distortion or other factors. The spatial transformation network is implemented by learning transformation parameters. The transformation process is usually achieved through an affine transformation matrix. Said, the formula is:

[0106] ;

[0107] in, , , , Control rotation and scaling, and Control translation. After affine transformation, the corrected feature map can make the shape of text and table more regular, which is helpful for subsequent processing. Perform adaptive binarization based on the corrected feature map. Separate the text area from the background in the image, and dynamically adjust the binarization result according to different brightness and contrast conditions by selecting an adaptive threshold to ensure that the text area can be accurately identified. Formulaically, the process of adaptive binarization can be expressed as follows:

[0108] ;

[0109] in, is the image in pixels The brightness value at is the threshold of adaptive calculation. If the brightness value exceeds the threshold, the pixel is classified as a text area, otherwise it is classified as a background area. Through this operation, a mask image of the text area is generated, and the part of the image containing text is clearly marked. Connected domain analysis is performed on the text area mask. Connected domain analysis is used to detect all connected pixel blocks in the image, that is, those pixel areas that are adjacent to each other and have the same value. Through this analysis, each independent text line area is identified to form a candidate text line area. Each candidate area corresponds to a possible line of text. In this way, each line of text in the image can be effectively segmented. Based on the candidate text line area, the corrected feature map is cropped, and each text line area is extracted separately to form a series of text line feature sequences. Each feature sequence corresponds to a line of text. These feature sequences not only contain local features of the text, but also retain global layout information. The text line feature sequence is input into the bidirectional long short-term memory network for context feature extraction. Bidirectional LSTM is a deep learning model that can capture the contextual information of sequence data and can extract context-aware feature representations. Through the formula:

[0110] ;

[0111] in, It is the current moment The hidden state of is the input at the current moment, is the hidden state of the previous moment. Through bidirectional LSTM, the model can provide a richer contextual information for each text character and more accurately understand the meaning of each character in the entire text line. After obtaining the context-aware feature representation, the attention weight is calculated to obtain the initial attention distribution. The role of the attention mechanism is to assign a weight value to each text character to determine the importance of the character in the current context. The formula is expressed as:

[0112] ;

[0113] in, It's time The attention weight, is the attention score calculated based on contextual features, is the total length of the sequence. Through this formula, the model is able to calculate the importance of each character and generate an initial attention distribution. In order to optimize the attention distribution, a temperature scaling operation is performed on the initial attention distribution. Temperature scaling strengthens or weakens the weights of certain characters by adjusting the smoothness of the attention score. The temperature scaling formula is:

[0114] ;

[0115] in, is the temperature parameter, the larger The smaller the value, the smoother the distribution. The value will make the distribution more sharp. Through this operation, the initial attention distribution is adjusted to make it more in line with the needs of the current task. A weighted sum operation is performed based on the adjusted attention distribution and the context-aware feature representation to obtain the context vector. The importance of each character is weighted through the attention mechanism to generate a comprehensive context representation. The formula for weighted summation is:

[0116] ;

[0117] in, is the context vector, is the adjusted attention weight, It is a context-aware feature representation. Through this operation, the model can obtain a context vector containing global information for text decoding. The generated context vector is input into the multi-layer perceptron network for processing. The function of the multi-layer perceptron is to classify the context vector and generate a probability distribution of character categories. Each output node represents a possible character category, and the output of the multi-layer perceptron can be expressed as:

[0118] ;

[0119] in, is the weight matrix of the multilayer perceptron, is the context vector, is the bias term, and finally the probability distribution of each character category is generated through the softmax function . These probability values ​​represent the possibility that each character belongs to a specific category. Based on the generated character category probability distribution, beam search decoding is performed to obtain the bill text data. Beam search is a heuristic search algorithm that can select the optimal path from multiple candidate decoding paths and generate the most likely text sequence. Through the above steps, the model can accurately extract the text from the bill image and convert it into structured text data.

[0120] In a specific embodiment, the process of executing step 600 may specifically include the following steps:

[0121] Merge the table structure data and the bill text data to obtain initial fused data, and perform field mapping on the initial fused data based on a predefined bill template to obtain structured bill data;

[0122] Perform named entity recognition on the entity information in the structured bill data to obtain a candidate entity list, and perform entity disambiguation on the candidate entity list based on the bill domain knowledge base to obtain a unique entity identifier;

[0123] Retrieve relevant attribute information from the bill domain knowledge base according to the unique entity identifier to obtain entity attribute data, and fuse the entity attribute data with the structured bill data to obtain enhanced bill data;

[0124] Based on the preset bill field integrity rules, the enhanced bill data is tested for missing values ​​to obtain a list of fields to be completed, and relevant completion information is retrieved from the bill domain knowledge base according to the list of fields to be completed to obtain candidate completion data;

[0125] Context relevance scoring is performed on the candidate completion data to obtain the optimal completion information, and the enhanced bill data is completed based on the optimal completion information to obtain a structured bill recognition result.

[0126] Specifically, the table structure data is effectively fused with the bill text data. The table structure data usually contains the layout and relationship of the cells in the table, while the bill text data contains the actual content in the bill. By merging these two types of data to form the initial fused data, the layout information and text content of the bill can be processed simultaneously under an overall framework. The initial fused data is field mapped based on the predefined bill template to obtain structured bill data. The bill template is a standardized structure that defines the fields in common bills, such as invoice number, date, amount, supplier name, etc. The field mapping process ensures that the data can be classified according to the expected structure by matching the information in the initial fused data with the standard fields in the bill template. Named entity recognition is performed on the entity information in the structured bill data to identify specific entities in the text, such as company name, address, date, amount, etc. Through the named entity recognition algorithm, a list of candidate entities is extracted from the structured bill data. These entity lists contain all potential key information in the bill. After identifying the candidate entity list, entity disambiguation is performed to ensure that the same entity can accurately correspond to a unique identifier in different contexts. In bill processing, some entities may be polysemous or ambiguous. For example, different companies may have the same name, or a date field may have different meanings in different scenarios. To solve this problem, the candidate entity list is disambiguated based on the bill domain knowledge base. The bill domain knowledge base contains a large number of entities and their related information, such as company registration number, address, etc. By querying the knowledge base, each candidate entity can be matched to obtain a unique entity identifier. Retrieve relevant attribute information from the bill domain knowledge base based on the unique entity identifier. Entity attribute information is supplementary information about the entity. For example, the attributes of a company name may include registered address, legal representative, etc. By retrieving these attribute information, the context information of each entity is enriched. These entity attribute data are fused with structured bill data to obtain enhanced bill data. The information fusion process improves the content of the bill by merging the attribute data of the entity with the basic data in the bill. Missing value detection is performed on the enhanced bill data. In the actual processing of bills, some fields may be missing. For example, an invoice may lack some mandatory fields. In order to ensure the integrity of the data, the enhanced bill data is detected based on the preset bill field integrity rules to identify which fields are missing. These rules usually define the mandatory or optional nature of each field in the bill. Through the detection step, a list of fields to be completed is generated, which contains the fields that need to be further filled. Based on the list of fields to be completed, relevant completion information is retrieved from the bill domain knowledge base. The bill domain knowledge base not only contains the basic information of the entity, but also may contain similar field data in historical bills. By querying these historical records or contextual information related to the current bill, candidate completion data is generated.In order to determine the optimal completion data, context relevance scoring is performed on the candidate data. Context relevance scoring is performed by calculating the degree of match between each candidate completion data and the current bill context. For example, the degree of match between the supplier name and address can be calculated by comparing the similarity of the name, geographical location and other information. Through this process, each candidate data is scored and the option with the highest score is selected as the optimal completion information. Based on the optimal completion information, the enhanced bill data is completed to complete the data supplement. By inserting the optimal completion information into the missing field, the complete bill data is generated. The final structured bill recognition result not only contains the original bill content, but also improves the information structure of the bill through entity recognition, attribute completion and other methods, making the data more complete and rich.

[0127] The above describes the bill scene recognition method in the embodiment of the present application. The following describes the bill scene recognition device 10 in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of the bill scene recognition device 10 includes:

[0128] The processing module 11 is used to perform multi-scale convolution processing on the bill image to obtain initial feature data;

[0129] A transformation module 12, used for performing local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set;

[0130] An extraction module 13 is used to perform multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features;

[0131] A construction module 14 is used to construct a graph structure of the bill table based on multi-scale fusion features, and perform node feature aggregation processing through iterative graph convolution to obtain table structure data;

[0132] The decoding module 15 is used to perform spatial transformation and bidirectional sequence encoding on the multi-scale fusion features, and perform decoding processing in combination with the dynamic attention mechanism to obtain the bill text data;

[0133] The integration module 16 is used to integrate, entity link and complete information of the table structure data and the bill text data based on the bill field knowledge base to obtain a structured bill recognition result.

[0134] Through the synergy of the above components, multi-scale convolution processing and data enhancement technology, the model's adaptability to various complex bill scenarios is improved, and the robustness of recognition is enhanced. The use of multi-layer feature extraction and fusion processing can effectively capture the different levels of features of bill images, and improve the recognition ability of complex structures and detailed information. The introduction of graph structure and iterative graph convolution technology significantly improves the understanding and information extraction capabilities of complex table structures. Combining spatial transformation and dynamic attention mechanism, the model's recognition ability for tilted and deformed text is enhanced, and the accuracy of text recognition is improved. By integrating the bill domain knowledge base, entity linking and information completion are achieved, which greatly improves the completeness and accuracy of the recognition results. This method has strong versatility and can be applied to various types of bill recognition tasks, reducing the need to develop specialized algorithms for different bill types. It improves the efficiency of the overall recognition process and is suitable for large-scale bill processing scenarios.

[0135] The present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the bill scene recognition method in the above-mentioned embodiments.

[0136] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the bill scene recognition method.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0139] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A bill scene recognition method, characterized in that: The method comprises: Perform multi-scale convolution processing on the bill image to obtain initial feature data; Performing local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set; Performing multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features; Constructing a graph structure of a bill table based on the multi-scale fusion features, and performing node feature aggregation processing through iterative graph convolution to obtain table structure data; Performing spatial transformation and bidirectional sequence encoding on the multi-scale fusion features, and decoding processing in combination with a dynamic attention mechanism to obtain bill text data; Based on the bill domain knowledge base, the table structure data and the bill text data are integrated, entity linked and information completed to obtain a structured bill recognition result.

2. The bill scene recognition method according to claim 1, characterized in that: The multi-scale convolution processing is performed on the bill image to obtain initial feature data, including: Preprocess the input bill image to obtain standardized image data; Inputting the standardized image data into a first convolution layer, performing a convolution operation using a 7×7 convolution kernel to obtain a first convolution feature map, and performing a maximum pooling operation on the first convolution feature map to obtain a downsampled feature map; Inputting the downsampled feature map into the second convolution layer, performing a convolution operation using a 5×5 convolution kernel to obtain a second convolution feature map, and performing an average pooling operation on the second convolution feature map to obtain a compressed feature map; Inputting the compressed feature map into a third convolutional layer, performing a convolution operation using a 3×3 convolution kernel to obtain a third convolutional feature map, and performing spatial pyramid pooling on the third convolutional feature map to obtain a multi-scale feature vector; Inputting the multi-scale feature vector into a fully connected layer for processing to obtain a feature representation, and performing a batch normalization operation on the feature representation to obtain a normalized feature; A nonlinear activation function operation is performed on the normalized features to obtain initial feature data.

3. The bill scene recognition method according to claim 1, characterized in that: The performing local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set includes: Randomly selecting an area of ​​the initial feature data to obtain an area to be transformed, and performing an affine transformation matrix analysis on the area to be transformed based on a preset transformation parameter range to obtain a local transformation matrix; Performing an affine transformation on the area to be transformed according to the local transformation matrix to obtain a local deformation feature, and fusing the local deformation feature with the initial feature data to obtain a global deformation feature; Based on a predefined illumination model, calculating illumination change parameters of the global deformation feature to obtain an illumination adjustment factor, and adjusting brightness of the global deformation feature according to the illumination adjustment factor to obtain a brightness adjustment feature; Based on Gaussian distribution, the brightness adjustment feature is subjected to local highlight and shadow mask processing to obtain a lighting effect mask; Performing local illumination enhancement on the brightness adjustment feature according to the illumination effect mask to obtain a simulated illumination feature, and performing contrast adjustment on the simulated illumination feature to obtain a contrast enhancement feature; A plurality of enhanced samples with different degrees are generated based on the contrast enhancement feature to obtain an enhanced data set.

4. The bill scene recognition method according to claim 1, characterized in that: The step of performing multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features includes: Inputting the enhanced data set into a shallow convolutional network for processing to obtain a low-level feature map, and performing spatial pyramid pooling based on the low-level feature map to obtain multi-scale low-level features; Inputting the low-level feature map into a middle-layer convolutional network for processing to obtain a middle-level feature map, and performing adaptive average pooling based on the middle-level feature map to obtain a fixed-dimensional middle-level feature; Inputting the intermediate feature map into a deep convolutional network for processing to obtain a high-level feature map, and performing global maximum pooling based on the high-level feature map to obtain global high-level features; Performing channel attention calculation on the multi-scale low-level features, the fixed-dimensional intermediate features, and the global high-level features to obtain weighted features, and performing feature interaction operations based on the weighted features to obtain interactive features; The interactive features are subjected to nonlinear transformation and normalization processing to obtain standardized fusion features, and residual connection and feature recalibration are performed based on the standardized fusion features to obtain multi-scale fusion features.

5. The bill scene recognition method according to claim 1, characterized in that: The graph structure of the bill table is constructed based on the multi-scale fusion features, and node feature aggregation processing is performed through iterative graph convolution to obtain table structure data, including: Performing edge detection on the multi-scale fusion features to obtain a table line feature map, and performing Hough transform based on the table line feature map to obtain a table cell candidate area; Performing non-maximum suppression processing on the table cell candidate area to obtain accurate table cell positions, and constructing an initial graph structure based on the accurate table cell positions to obtain a table topology graph; Input each node in the table topology graph into the regional feature extraction network for feature extraction to obtain the initial feature of the node; Calculating a similarity matrix between nodes based on the initial features of the nodes to obtain an adjacency matrix of the graph, and performing a graph attention mechanism analysis on the adjacency matrix of the graph to obtain a weighted adjacency matrix; Performing a graph convolution operation based on the weighted adjacency matrix and the initial node features to obtain updated node features, and performing multiple iterative graph convolution operations on the updated node features to obtain a final node representation; Graph structure decoding is performed based on the final node representation to obtain table structure data including cell type and content.

6. The bill scene recognition method according to claim 1, characterized in that: The multi-scale fusion features are spatially transformed and bidirectionally sequenced, and decoded in combination with a dynamic attention mechanism to obtain bill text data, including: Performing a spatial transformation network on the multi-scale fusion feature to obtain a corrected feature map, and performing adaptive binarization processing based on the corrected feature map to obtain a text area mask; Performing a connected domain analysis on the text region mask to obtain a candidate text line region, and cropping the corrected feature map based on the candidate text line region to obtain a text line feature sequence; Inputting the text line feature sequence into a bidirectional LSTM network to extract context features to obtain a context-aware feature representation, and calculating attention weights based on the context-aware feature representation to obtain an initial attention distribution; Performing a temperature scaling operation on the initial attention distribution to obtain an adjusted attention distribution, and performing a weighted summation based on the adjusted attention distribution and the context-aware feature representation to obtain a context vector; The context vector is input into a multi-layer perceptron network for processing to obtain a character category probability distribution, and beam search decoding is performed based on the character category probability distribution to obtain bill text data.

7. The bill scene recognition method according to claim 1, characterized in that: The table structure data and the bill text data are integrated, entity linked and information completed based on the bill domain knowledge base to obtain a structured bill recognition result, including: Merging the table structure data and the bill text data to obtain initial fused data, and performing field mapping on the initial fused data based on a predefined bill template to obtain structured bill data; Perform named entity recognition on the entity information in the structured bill data to obtain a candidate entity list, and perform entity disambiguation on the candidate entity list based on the bill domain knowledge base to obtain a unique entity identifier; Retrieving relevant attribute information from the bill domain knowledge base according to the unique entity identifier to obtain entity attribute data, and fusing the entity attribute data with the structured bill data to obtain enhanced bill data; Perform missing value detection on the enhanced bill data based on preset bill field integrity rules to obtain a list of fields to be completed, and retrieve relevant completion information from the bill domain knowledge base according to the list of fields to be completed to obtain candidate completion data; Context relevance scoring is performed on the candidate completion data to obtain optimal completion information, and information completion is performed on the enhanced bill data based on the optimal completion information to obtain a structured bill recognition result.

8. A bill scene recognition device, characterized in that: The device is used to perform the bill scene recognition method according to any one of claims 1 to 7, comprising: A processing module, used for performing multi-scale convolution processing on the bill image to obtain initial feature data; A transformation module, used for performing local affine transformation and illumination simulation on the initial feature data to obtain an enhanced data set; An extraction module, used for performing multi-layer feature extraction and fusion processing on the enhanced data set to obtain multi-scale fusion features; A construction module is used to construct a graph structure of the bill table based on the multi-scale fusion features, and perform node feature aggregation processing through iterative graph convolution to obtain table structure data; A decoding module, used to perform spatial transformation and bidirectional sequence encoding on the multi-scale fusion features, and perform decoding processing in combination with a dynamic attention mechanism to obtain bill text data; The integration module is used to integrate, entity link and complete the information of the table structure data and the bill text data based on the bill field knowledge base to obtain a structured bill recognition result.

9. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the bill scene recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the bill scene recognition method according to any one of claims 1 to 7 is implemented.

Images