Electronic certificate classification method based on multi-modal information fusion

Abstract

The invention relates to an electronic certificate classification method based on multi-modal information fusion. The method comprises the following steps: collecting image data of electronic certificates, preprocessing the image data, and constructing an electronic certificate image data set; extracting character data in the electronic certificate image data set, preprocessing the character data, and constructing an electronic certificate character data set; constructing an electronic certificate image feature extraction model, and inputting the electronic certificate image data set into the electronic certificate image feature extraction model to obtain an electronic certificate image feature set; constructing an electronic certificate character feature extraction model, and inputting the electronic certificate character data set into the electronic certificate character feature extraction model to obtain an electronic certificate character feature set; splicing the electronic certificate image feature set and the electronic certificate character feature set to obtain an electronic certificate fusion feature set; and constructing an electronic certificate classification model, training the electronic certificate classification model based on the electronic certificate fusion feature set, and performing classification through the trained electronic certificate classification model.

Description

Technical Field

[0001] The present invention relates to an electronic voucher classification method based on multimodal information fusion, and belongs to the technical field of voucher data management. Background Art

[0002] In the current information processing and management industry, enterprises and organizations generate a large number of electronic documents (such as invoices, receipts, contracts, etc.) every day.

[0003] In order to effectively manage and utilize these credentials, automated credential classification systems are becoming increasingly important. Currently, existing solutions on the market mainly rely on single-modal data processing methods, such as extracting text information from images based on OCR (optical character recognition) technology, or directly performing natural language processing on text content to achieve classification.

[0004] Although these methods perform well in specific scenarios, they often fail to fully exploit all types of information contained in the credential, especially when the credential contains both important image and text elements.

[0005] Traditional methods typically focus on a single type of input (e.g., only text or only images), while ignoring other data sources that may provide additional contextual information. This results in limited classification accuracy, especially when dealing with complex credential types.

[0006] Due to the huge differences in formats, languages, and structures of different credentials, existing classification algorithms are difficult to adapt to diverse application scenarios, and their performance is easily affected by external factors, such as poor image quality or complex text layout. Summary of the Invention

[0007] In order to solve the above problems existing in the prior art, the present invention proposes an electronic certificate classification method based on multimodal information fusion.

[0008] The technical solutions of the present invention are as follows: In one aspect, the present invention provides an electronic credential classification method based on multimodal information fusion, comprising the following steps: Collect the image data of electronic vouchers and construct an electronic voucher image dataset after preprocessing; Extract text data from the electronic voucher image dataset and construct it into an electronic voucher text dataset after preprocessing; Construct an electronic voucher image feature extraction model, and input the electronic voucher image dataset into the electronic voucher image feature extraction model to obtain an electronic voucher image feature set; Construct an electronic voucher text feature extraction model, and input the electronic voucher text dataset into the electronic voucher text feature extraction model to obtain an electronic voucher text feature set; The electronic voucher image feature set and the electronic voucher text feature set are spliced ​​together to obtain the electronic voucher fusion feature set; Construct an electronic credential classification model, train the electronic credential classification model based on the electronic credential fusion feature set, and classify electronic credentials using the trained electronic credential classification model.

[0009] Preferably, the image data of the collected electronic voucher is cleaned and then constructed into an electronic voucher image dataset.

[0010] Preferably, the text data of each sample in the electronic voucher image dataset is extracted based on the RapidOCR tool and preprocessed to construct an electronic voucher text dataset.

[0011] Preferably, the electronic voucher image feature extraction model is constructed based on the EfficientNetV2-S network and the RegNetY-8GF network; The electronic voucher image dataset is input into the EfficientNetV2-S network, and the EfficientNetV2-S network outputs a first electronic voucher image feature set; The electronic voucher image dataset is input into the RegNetY-8GF network, and the RegNetY-8GF network outputs a second electronic voucher image feature set; Constructing a dimensionally unified convolutional layer, inputting the first electronic credential image feature set and the second electronic credential image feature set into the dimensionally unified convolutional layer respectively to obtain the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set, and then performing global average pooling on the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set to extract global statistical information of the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set; The weight vector is generated by a multi-layer perceptron based on the global statistical information of the first electronic voucher image feature set and the second electronic voucher image feature set with unified dimensions, as shown in the following formula: ; ; in: a weight vector representing a first electronic voucher image feature set having uniform dimensions; Global statistical information representing a dimensionally unified feature set of the first electronic voucher image; a weight vector representing a second electronic voucher image feature set having uniform dimensions; Global statistical information representing a dimensionally unified second electronic credential image feature set; represents the softmax activation function; represents a multilayer perceptron; The first electronic voucher image feature set and the second electronic voucher image feature set with unified dimensions are weightedly summed by their corresponding weight vectors to obtain a fused electronic voucher image feature set, and the fused electronic voucher image feature set is used as the output result of the electronic voucher image feature extraction model.

[0012] Preferably, the convolution kernel size of the dimensionally unified convolution layer is ; The dimension unification convolution layer unifies the dimensions of the first electronic voucher image feature set and the second electronic voucher image feature set to 512 dimensions.

[0013] Preferably, a spatial attention module is provided after the backbone network output layer of the EfficientNetV2-S network and the RegNetY-8GF network; For any network, the spatial attention module includes a feature aggregation layer and a convolutional activation layer; The feature aggregation layer is used to calculate the channel mean and maximum value of all feature maps output by the backbone network output layer, and construct the channel mean feature map and maximum feature map of a single channel based on the channel mean and maximum value of all feature maps, and then splice the channel mean feature map and maximum feature map with all feature maps output by the backbone network output layer to obtain an aggregated feature set; The convolution activation layer is used to convolve the aggregated feature set, generate normalized attention weights through the sigmoid activation function, assign the attention weights to the aggregated feature set, and obtain the output of the current network.

[0014] Preferably, the convolution kernel size of the convolution activation layer is .

[0015] Preferably, the electronic voucher text feature extraction model is constructed based on a pre-trained BERT model.

[0016] Preferably, the electronic certificate image feature set and the electronic certificate text feature set are concatenated and then linearly transformed, and then the electronic certificate fusion feature set is generated through the ReLU activation function.

[0017] Preferably, the electronic voucher classification model consists of a Dropout layer and a fully connected layer.

[0018] The present invention has the following beneficial effects: 1. By fusing image features with text features, the present invention fully exploits the complementarity of image and text information in electronic credentials, significantly improving the accuracy and robustness of the classification model compared to methods that rely solely on a single modality.

[0019] 2. The present invention combines image features and text semantic features to better handle voucher types with inconsistent formats, complex content, and diverse layouts, and is suitable for a wider range of practical application scenarios.

[0020] 3. The present invention introduces a spatial attention module into the image feature extraction network, enabling the model to automatically focus on discriminative areas in the image, further improving the effectiveness of feature expression. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are to be executed.

[0024] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0026] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] Example 1: See also Figure 1 , an electronic voucher classification method based on multimodal information fusion, comprising the following steps: Collect the image data of electronic vouchers and construct an electronic voucher image dataset after preprocessing; Extract text data from the electronic voucher image dataset and construct it into an electronic voucher text dataset after preprocessing; Construct an electronic voucher image feature extraction model, and input the electronic voucher image dataset into the electronic voucher image feature extraction model to obtain an electronic voucher image feature set; Construct an electronic voucher text feature extraction model, and input the electronic voucher text dataset into the electronic voucher text feature extraction model to obtain an electronic voucher text feature set; The electronic voucher image feature set and the electronic voucher text feature set are spliced ​​together to obtain the electronic voucher fusion feature set; Construct an electronic credential classification model, train the electronic credential classification model based on the electronic credential fusion feature set, and classify electronic credentials using the trained electronic credential classification model.

[0028] As a preferred implementation of this embodiment, the image data of the electronic voucher collected includes: electronic invoices (ordinary invoices), electronic invoices (VAT special invoices), blockchain invoices, bank electronic statements, bank electronic receipts, VAT electronic ordinary invoices, VAT electronic special invoices, paper-electronic VAT ordinary invoices, and paper-electronic VAT special invoices; Clean the above image data and delete non-credential data and low-quality images; After data cleaning, data enhancement is performed on each category of image data. The specific enhancement methods include: flipping, rotation, wave distortion, Gaussian noise addition, and random occlusion; An electronic voucher image dataset is constructed based on image data of all categories after data enhancement. In this embodiment, each category in the electronic voucher image dataset contains 2,000 images, for a total of 18,000 training data. The dataset is divided into training set and test set in a ratio of 7:3.

[0029] As a preferred implementation of this embodiment, the text data of each sample in the electronic voucher image dataset is extracted based on the RapidOCR tool and preprocessed to construct an electronic voucher text dataset.

[0030] As a preferred implementation of this embodiment, the electronic voucher image feature extraction model is constructed based on the EfficientNetV2-S network and the RegNetY-8GF network; The electronic voucher image dataset is input into the EfficientNetV2-S network, and the EfficientNetV2-S network outputs a first electronic voucher image feature set; The electronic voucher image dataset is input into the RegNetY-8GF network, and the RegNetY-8GF network outputs a second electronic voucher image feature set; Constructing a dimensionally unified convolutional layer, inputting the first electronic credential image feature set and the second electronic credential image feature set into the dimensionally unified convolutional layer respectively to obtain the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set, and then performing global average pooling on the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set to extract global statistical information of the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set; The weight vector is generated by a multi-layer perceptron based on the global statistical information of the first electronic voucher image feature set and the second electronic voucher image feature set with unified dimensions, as shown in the following formula: ; ; in: a weight vector representing a first electronic voucher image feature set having uniform dimensions; Global statistical information representing a dimensionally unified feature set of the first electronic voucher image; a weight vector representing a second electronic voucher image feature set having uniform dimensions; Global statistical information representing a dimensionally unified second electronic credential image feature set; represents the softmax activation function; represents a multilayer perceptron; The first electronic voucher image feature set and the second electronic voucher image feature set with unified dimensions are weightedly summed by their corresponding weight vectors to obtain a fused electronic voucher image feature set, and the fused electronic voucher image feature set is used as the output result of the electronic voucher image feature extraction model.

[0031] As a preferred implementation of this embodiment, the convolution kernel size of the dimensionally unified convolution layer is ; The dimension unification convolution layer unifies the dimensions of the first electronic voucher image feature set and the second electronic voucher image feature set to 512 dimensions.

[0032] As a preferred implementation of this embodiment, a spatial attention module is provided after the backbone network output layer of the EfficientNetV2-S network and the RegNetY-8GF network; For any network, the spatial attention module includes a feature aggregation layer and a convolutional activation layer; The feature aggregation layer is used to calculate the channel mean and maximum value of all feature maps output by the backbone network output layer, and construct the channel mean feature map and maximum feature map of a single channel based on the channel mean and maximum value of all feature maps, and then splice the channel mean feature map and maximum feature map with all feature maps output by the backbone network output layer to obtain an aggregated feature set; The convolution activation layer is used to convolve the aggregated feature set, generate normalized attention weights through the sigmoid activation function, assign the attention weights to the aggregated feature set, and obtain the output of the current network.

[0033] As a preferred implementation of this embodiment, the convolution kernel size of the convolution activation layer is .

[0034] As a preferred implementation of this embodiment, the electronic voucher text feature extraction model is constructed based on the pre-trained BERT model. In this embodiment, the maximum sequence length of the BERT model's word segmenter is set to 256, which means that when processing the input text, the maximum length of each input sample is limited to 256 tokens. If the input text exceeds this length, it will be truncated to adapt to the set maximum length; if the text length is insufficient, it will usually be padded to 256 tokens to ensure that all input data has a consistent size.

[0035] As a preferred implementation method of this embodiment, the electronic certificate image feature set and the electronic certificate text feature set are spliced ​​and then linearly transformed (such as scaling, rotation or projection operations, which help improve the model's ability to understand complex certificates and enhance the model's generalization ability). After the linear transformation, the spliced ​​data set is passed through the ReLU activation function to generate an electronic certificate fusion feature set.

[0036] As a preferred implementation of this embodiment, the electronic voucher classification model is composed of a Dropout layer and a fully connected layer; The Dropout layer can force the network to learn more robust features because it can randomly discard some neurons; In this embodiment, the loss function of the electronic voucher classification model is the cross-entropy loss function; the optimizer: a stochastic gradient descent (SGD) optimizer with a momentum of 0.9 is used; the initial learning rate is set to 0.0001, which has been verified through multiple experiments to ensure the convergence speed while avoiding instability caused by excessive update step size; the batch size and the number of iterations are set to 64 and 32, respectively.

[0037] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0038] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0039] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0040] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), magnetic disk or optical disk, and other media that can store program code.

[0041] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An electronic voucher classification method based on multimodal information fusion, characterized in that: The following steps are involved: Collect the image data of electronic vouchers and construct an electronic voucher image dataset after preprocessing; Extract text data from the electronic voucher image dataset and construct it into an electronic voucher text dataset after preprocessing; Construct an electronic voucher image feature extraction model, and input the electronic voucher image dataset into the electronic voucher image feature extraction model to obtain an electronic voucher image feature set; Construct an electronic voucher text feature extraction model, and input the electronic voucher text dataset into the electronic voucher text feature extraction model to obtain an electronic voucher text feature set; The electronic voucher image feature set and the electronic voucher text feature set are spliced ​​together to obtain the electronic voucher fusion feature set; Construct an electronic credential classification model, train the electronic credential classification model based on the electronic credential fusion feature set, and classify electronic credentials using the trained electronic credential classification model.

2. The electronic voucher classification method based on multimodal information fusion according to claim 1 is characterized in that: The image data of the collected electronic voucher is cleaned and then constructed into an electronic voucher image data set.

3. The electronic voucher classification method based on multimodal information fusion according to claim 1 is characterized in that: Based on the RapidOCR tool, the text data of each sample in the electronic voucher image dataset is extracted and preprocessed to construct an electronic voucher text dataset.

4. The electronic voucher classification method based on multimodal information fusion according to claim 1 is characterized in that: The electronic voucher image feature extraction model is built based on the EfficientNetV2-S network and the RegNetY-8GF network; The electronic voucher image dataset is input into the EfficientNetV2-S network, and the EfficientNetV2-S network outputs a first electronic voucher image feature set; The electronic voucher image dataset is input into the RegNetY-8GF network, and the RegNetY-8GF network outputs a second electronic voucher image feature set; Constructing a dimensionally unified convolutional layer, inputting the first electronic credential image feature set and the second electronic credential image feature set into the dimensionally unified convolutional layer respectively to obtain the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set, and then performing global average pooling on the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set to extract global statistical information of the dimensionally unified first electronic credential image feature set and the second electronic credential image feature set; The weight vector is generated by a multi-layer perceptron based on the global statistical information of the first electronic voucher image feature set and the second electronic voucher image feature set with unified dimensions, as shown in the following formula: ； ； in: a weight vector representing a first electronic voucher image feature set having uniform dimensions; Global statistical information representing a dimensionally unified feature set of the first electronic voucher image; a weight vector representing a second electronic voucher image feature set having uniform dimensions; Global statistical information representing a dimensionally unified second electronic credential image feature set; represents the softmax activation function; represents a multilayer perceptron; The first electronic voucher image feature set and the second electronic voucher image feature set with unified dimensions are weightedly summed by their corresponding weight vectors to obtain a fused electronic voucher image feature set, and the fused electronic voucher image feature set is used as the output result of the electronic voucher image feature extraction model.

5. The electronic voucher classification method based on multimodal information fusion according to claim 4 is characterized in that: The convolution kernel size of the dimension-unified convolution layer is ; The dimension unification convolution layer unifies the dimensions of the first electronic voucher image feature set and the second electronic voucher image feature set to 512 dimensions.

6. The electronic voucher classification method based on multimodal information fusion according to claim 4 is characterized in that: The EfficientNetV2-S network and the RegNetY-8GF network are both equipped with a spatial attention module after the backbone network output layer; For any network, the spatial attention module includes a feature aggregation layer and a convolutional activation layer; The feature aggregation layer is used to calculate the channel mean and maximum value of all feature maps output by the backbone network output layer, and construct the channel mean feature map and maximum feature map of a single channel based on the channel mean and maximum value of all feature maps, and then splice the channel mean feature map and maximum feature map with all feature maps output by the backbone network output layer to obtain an aggregated feature set; The convolution activation layer is used to convolve the aggregated feature set, generate normalized attention weights through the sigmoid activation function, assign the attention weights to the aggregated feature set, and obtain the output of the current network.

7. The electronic voucher classification method based on multimodal information fusion according to claim 6 is characterized in that: The convolution kernel size of the convolution activation layer is .

8. The electronic voucher classification method based on multimodal information fusion according to claim 1 is characterized in that: The electronic voucher text feature extraction model is built based on the pre-trained BERT model.

9. The electronic voucher classification method based on multimodal information fusion according to claim 1 is characterized in that: The electronic certificate image feature set and the electronic certificate text feature set are concatenated and linearly transformed, and then the electronic certificate fusion feature set is generated through the ReLU activation function.

10. The electronic voucher classification method based on multimodal information fusion according to claim 1 is characterized in that: The electronic voucher classification model consists of a Dropout layer and a fully connected layer.

Images