Method, system, device and medium for financial form recognition

Abstract

This invention relates to a method, system, device, and medium for recognizing financial forms. The method includes: acquiring a financial form to be recognized; extracting feature maps of different scales from the financial form using a segmentation-based text detection algorithm and fusing them; identifying and extracting text regions from the fused feature maps; extracting text sequence features from the text regions and transcribing the extracted text sequence features based on an attention mechanism to extract text sequences from the text regions; encoding and semantically enhancing the text sequences to obtain a comprehensive vector sequence; and decoding the comprehensive vector sequence to recognize named entities in the financial form. This invention achieves automated, refined extraction and structured processing of financial form information.

Description

Technical Field

[0001] This invention relates to the field of financial information extraction technology, and in particular to a method, system, device and medium for identifying financial forms. Background Technology

[0002] With the rapid development of the financial industry and technological advancements, the demand for digital financial services is constantly growing. Faced with increasingly sophisticated financial systems, improving overall operational efficiency and risk management to ensure user asset security is paramount. Therefore, recording and analyzing user transaction behavior, market environment changes, and risk warning information has become crucial when processing financial data. The collection and processing of this information relies on image recognition technology, combined with deep learning algorithms and computer vision technology. This enables accurate identification and analysis of information in different financial scenarios, providing real-time financial market data around the clock and playing a key role in risk control and early warning.

[0003] However, current form recognition technologies in the financial field generally employ traditional template matching schemes, rule-based text extraction schemes, or hybrid feature classification schemes. These schemes lack flexibility, have high maintenance costs, and struggle to adapt to the diversity and variability of financial forms, thus failing to adequately meet the needs of financial form processing. Therefore, there is a need to provide a method, system, device, and medium for recognizing financial forms. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, system, device and medium for identifying financial forms, which improves the problem of low accuracy in the prior art in extracting nested named entities and identifying their semantic relationships from complex and ever-changing financial forms.

[0005] To achieve the above and other related objectives, this invention provides a method for identifying financial forms, comprising: acquiring a financial form to be identified; extracting feature maps of different scales from the financial form and fusing them according to a segmentation-based text detection algorithm, identifying and extracting text regions from the fused feature maps; extracting text sequence features from the text regions, and transcribing the extracted text sequence features based on an attention mechanism to extract text sequences from the text regions; encoding and semantically enhancing the text sequences to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify named entities in the financial form.

[0006] In one embodiment of the present invention, the step of extracting feature maps of different scales from the financial form and fusing them according to a segmentation-based text detection algorithm, and identifying and extracting text regions from the fused feature map, includes: inputting the financial form into a residual network, performing feature processing through each convolutional layer group to generate feature maps of different scales; wherein the residual network includes multiple cascaded convolutional layer groups, each convolutional layer group generating a feature map of one scale; performing feature enhancement on the smallest scale feature map, and arranging the enhanced smallest scale feature map and the remaining feature maps of each scale in descending order of scale; fusing the feature maps according to the arrangement order based on a feature pyramid algorithm to obtain a fused feature map; and identifying and extracting text regions from the fused feature map based on a differential binarization text detection algorithm.

[0007] In one embodiment of the present invention, the convolution at a preset position in the residual network is a deformable convolution.

[0008] In one embodiment of the present invention, the process of generating a feature map of the corresponding scale for the first convolutional layer group of the residual network includes: inputting the financial form into the convolutional module of the first convolutional layer group to obtain an initial feature map; inputting the initial feature map into the channel attention module of the first convolutional layer group, performing global max pooling and global average pooling on the financial form respectively, and processing the two pooled feature maps based on a shared multilayer perceptron to obtain a channel-weighted feature map; inputting the channel-weighted feature map into the spatial attention module of the first convolutional layer group, performing global max pooling and global average pooling on the channel-weighted feature map respectively and concatenating them, performing dilated convolution on the concatenated feature map, and fusing the dilated convolution feature map with the channel-weighted feature map to obtain a spatially weighted feature map; and inputting the spatially weighted feature map into the residual module of the first convolutional layer group to generate a feature map of the corresponding scale.

[0009] In one embodiment of the present invention, the process of generating feature maps of corresponding scales for each remaining convolutional layer group of the residual network includes: inputting the feature map generated by the previous convolutional layer group into the convolutional module of the current convolutional layer group to obtain an initial feature map of the current convolutional layer group at the corresponding scale; and inputting the initial feature map into the residual module of the current convolutional layer group to generate a feature map of the corresponding scale.

[0010] In one embodiment of the present invention, the step of extracting text sequence features from the text region and transcribing the extracted text sequence features based on an attention mechanism to extract the text sequence in the text region includes: extracting features from the text region based on a convolutional neural network, and transforming the extracted features based on context information using a bidirectional long short-term memory network to obtain a feature sequence; and transcribing the feature sequence based on the bidirectional long short-term memory network and an attention mechanism to extract the text sequence in the text region.

[0011] In one embodiment of the present invention, the step of encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify named entities in the financial form, includes: inputting the text sequence into a pre-trained language model to obtain an embedding vector sequence; capturing the context features of the text sequence based on a bidirectional long short-term memory network, encoding the context features to obtain a context vector sequence; selecting similar vector combinations of each context vector in the context vector sequence from a word vector library based on cosine similarity; weighting the similar vectors in the similar vector combinations according to an attention mechanism to obtain a semantically enhanced vector for each context vector, forming a semantically enhanced vector sequence of the context vector sequence; concatenating the semantically enhanced vector sequence, the context vector sequence, and the embedding vector sequence to obtain a comprehensive vector sequence; and decoding the comprehensive vector sequence based on a multi-layer entity recognizer to identify and extract nested named entities in the financial form; wherein each layer of the entity recognizer includes a bidirectional long short-term memory network and a dual-kernel convolutional neural network.

[0012] In one embodiment of the present invention, the named entity includes a semantic tag and entity content. After encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify the named entities in the financial form, the method further includes: inputting each named entity into a feedforward neural network for feature extraction to generate semantic tag features and content features corresponding to each named entity; combining the semantic tag features and entity content features of each named entity to obtain entity pair features of each named entity; inputting the entity pair features of each named entity into a dual affine classifier to obtain a relationship score between each named entity; and for each named entity: selecting the other named entity with the highest relationship score as the relationship named entity of the current named entity to form a relationship entity pair.

[0013] In one embodiment of the present invention, a financial form recognition system is also provided. The system includes: a form acquisition module for acquiring a financial form to be recognized; a text detection module for extracting feature maps of different scales from the financial form and performing fusion processing based on a segmentation-based text detection algorithm, and identifying and extracting text regions in the financial form from the fused feature maps; a text recognition module for extracting text sequence features in the text regions, and performing transcription processing on the extracted text sequence features based on an attention mechanism to extract text sequences in the text regions; and a financial nested named entity recognition module for encoding and semantically enhancing the text sequences to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify named entities in the financial form.

[0014] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device enables the financial form recognition method described in any of the preceding claims.

[0015] In one embodiment of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a computer processor, causes the computer to perform the financial form recognition method described in any of the preceding claims.

[0016] As described above, the financial form recognition method, system, device, and medium of the present invention have the following beneficial effects: After acquiring the financial form to be recognized, a segmentation-based text detection algorithm is used to extract and fuse multi-scale feature maps, improving the detection accuracy for text regions of different scales and complex backgrounds. Text sequence features of the text regions are extracted and transcribed using an attention mechanism, enhancing the accuracy of text content recognition. The text sequence is encoded and semantically enhanced to obtain a comprehensive vector sequence, which is then decoded to successfully identify named entities in the financial form. The present invention achieves automated, refined extraction and structured processing of financial form information. For images of complex financial forms, text recognition and extraction, key named entities and their relationships are extracted to form a unified structured output. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a financial form recognition method provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of a financial form;

[0019] Figure 3 This is a network structure diagram for extracting text regions from financial forms according to the present invention;

[0020] Figure 4 A network structure diagram for generating text regions for the first and remaining convolutional layer groups of this invention;

[0021] Figure 5 This is a network structure diagram for extracting text sequences from text regions according to the present invention;

[0022] Figure 6 This is a network structure diagram for extracting named entity sets from text sequences according to the present invention;

[0023] Figure 7 This is a network structure diagram for extracting entity relationships from a named entity set according to the present invention.

[0024] Figure 8 The diagram shown is a structural block diagram of a financial form recognition system provided in an embodiment of the present invention.

[0025] Figure 9 The diagram shown is a structural schematic of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0026] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0028] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0029] The inventors discovered that traditional manual processing methods are no longer sufficient to meet the growing demands for form processing in the financial sector, given the complexity of transaction data and customer information. Therefore, automating form processing using deep learning technology has become crucial. Visual solutions for financial forms have higher requirements than general text recognition tasks. Financial forms require more sophisticated data features; for example, specific elements such as numbers, dates, and barcodes need to be accurately extracted and identified. Furthermore, financial forms are often processed in challenging environments, including low light and numerous influencing factors. Therefore, using deep learning technology to acquire panoramic images and identify feature points in low-light and complex environments yields better results, effectively recording anomalies and providing more reliable data support. Traditional template matching schemes require pre-compiled templates, which are ineffective for complex form structures and frequently changing data formats. Rule-based text extraction schemes rely on manually written rules, requiring continuous adjustments and updates for specific form types and new forms, resulting in high maintenance costs. Hybrid feature classification schemes typically require large amounts of labeled data and feature engineering, making them ill-suited for the diversity and variability of financial forms, and the resulting data has limited practical value for financial institutions' risk management and security. It improves upon the problem of low accuracy in existing technologies for extracting nested named entities and identifying their semantic relationships from complex and ever-changing financial forms.

[0030] This invention provides a method for recognizing financial forms. After acquiring the financial form to be recognized, a segmentation-based text detection algorithm is used to extract and fuse multi-scale feature maps, improving the detection accuracy for text regions with different scales and complex backgrounds. Text sequence features of the text regions are extracted and transcribed using an attention mechanism, enhancing the accuracy of text content recognition. The text sequence is encoded and semantically enhanced to obtain a comprehensive vector sequence, which is then decoded to successfully identify named entities in the financial form. This invention achieves automated, refined extraction and structured processing of financial form information. For images of complex financial forms, text recognition and extraction, key named entities and their relationships are extracted, forming a unified structured output.

[0031] Please see Figure 1 The method for recognizing financial forms includes the following steps:

[0032] S1. Obtain the financial form to be identified.

[0033] Financial forms refer to structured or semi-structured documents used to record and transmit information related to financial transactions. Common types of financial forms include, but are not limited to, bank account opening applications, loan applications, insurance policies, financial statements, and stock transaction records. It should be noted that the method described in this invention can be applied to the identification and association of both printed and handwritten financial form content. For handwritten financial forms, an image of the form to be identified can be obtained by taking a photograph. For printed financial forms, an image of the form to be identified can be obtained by scanning or uploading an electronic document. Figure 2 The image shown is an image of a handwritten financial form (i.e., a loan application form). Since handwriting recognition is difficult, relying solely on manual identification and statistics is time-consuming and labor-intensive. Therefore, this method allows for the identification of relationships between various named entities. For example, from... Figure 2 One type of relationship can be identified as: Contact person's name: Zhang San, and their document type: ID card.

[0034] S2. Based on the segmentation-based text detection algorithm, feature maps of different scales are extracted from the financial form and fused. The text regions in the financial form are then identified and extracted from the fused feature maps.

[0035] This invention employs a segmentation-based text detection algorithm (DBNet), which performs multi-level convolutions on financial forms through multiple cascaded convolutional groups to extract feature maps at different scales. These feature maps are then gradually merged in a top-down manner through feature fusion to enhance the overall feature representation capability. Because the fused feature map contains both local and global information, the identified text regions are more accurate, thus improving the accuracy of subsequent text recognition.

[0036] In one embodiment of the present invention, the step of extracting feature maps of different scales from the financial form according to a segmentation-based text detection algorithm and performing fusion processing, and identifying and extracting text regions from the fused feature maps, includes:

[0037] The financial form is input into a residual network, and feature processing is performed through each convolutional layer group to generate feature maps of different scales; wherein, the residual network includes multiple cascaded convolutional layer groups, and each convolutional layer group generates a feature map of one scale;

[0038] Feature enhancement is performed on the feature map at the smallest scale, and the enhanced feature map at the smallest scale and the feature maps at the remaining scales are arranged in descending order of scale.

[0039] Based on the feature pyramid algorithm, the feature maps are fused according to their arrangement order to obtain the fused feature map.

[0040] Based on the differential binarization text detection algorithm, the text region in the financial form is identified and extracted from the fused feature map.

[0041] like Figure 3 As shown, a financial form image is input into a residual network (such as ResNet50). Since this residual network consists of multiple cascaded convolutional layers, each convolutional layer performs feature processing on the input information, extracting features to generate a feature map at a preset scale. Furthermore, to further improve the detection and recognition capability of text regions, the convolutions at preset positions in the residual network are deformable convolutions. Specifically, this invention also replaces all 3×3 convolutional layers (Conv3, Conv4, and Conv5) in the residual network with deformable convolutions. Because deformable convolutions can adaptively adjust the sampling position of the convolution kernel, they can more accurately locate the detected text regions, thereby improving the flexibility of feature extraction. Through the above operations, feature map sets of different scales output by the residual network (such as...) can be generated. Figure 3 In the model {C2, C3, C4, C5}, each feature map corresponds to a scale. This multi-scale feature extraction allows for better capture of text regions of different sizes and shapes in financial form images, thereby improving the accuracy of subsequent text recognition. Furthermore, to enrich the semantic information of high-level features and enhance the network's ability to perceive complex text regions, thus improving the accuracy of text region detection, this invention also adds a feature enhancement algorithm (FFA) to the smallest-scale feature map for feature enhancement.

[0042] Among them, the feature map of the smallest scale is the feature map obtained by the highest layer of the residual network (e.g., Figure 3 In C5). Specifically, the minimum-scale feature map generated by the convolutional layer group is first subjected to channel dimensionality reduction processing. By performing a 1×1 convolution on the feature map, the number of channels of the feature map is reduced from high dimension to 256 dimensions, thereby greatly reducing the amount of computation. Random pooling processing is performed on the dimensionality-reduced feature map to obtain multiple pooled feature maps. Specifically, three feature maps with different scales and rich contextual semantic information {α1=0.1,α2-0.2,α3=0.3} can be formed. For each of the above pooled feature maps, bilinear upsampling processing is performed to restore its size to the same size as the original feature map (i.e., C5). The feature maps after upsampling processing are fused and the enhanced minimum-scale feature map is obtained by using the sigmoid activation function. As shown in formula (1):

[0043]

[0044] Where Pool(C5,α) i ) indicates that α is applied to C5. i downsampling operation, F 1×1 Represents a 1x1 convolution, UP(·,α) i ) indicates that α is applied to the feature map. i The bilinear upsampling is performed, and FAA(C5) represents the enhanced minimum-scale feature map. The enhanced minimum-scale feature map and the remaining feature maps at each scale {C2, C3, C4} are arranged in descending order of scale (from largest to smallest), resulting in the arrangement {C5′, C4, C3, C2}.

[0045] Then, based on the feature pyramid network, starting from the smallest scale feature map, the feature maps are upsampled layer by layer to match their size with the next layer's high-resolution feature map. At each scale, the upsampled feature map Mi+1 is laterally concatenated with the corresponding high-resolution feature map Mi (i.e., the two feature maps are added element-wise), fusing high-level semantic information and low-level spatial detail information. Finally, the laterally concatenated feature maps are fused together, resulting in a final fused feature map.

[0046] The fused feature map is input into the DB text detection algorithm to predict and generate a probability map and a threshold map. A differentiable binarization function is then used to convert the probability map and threshold map into a binary image, thereby accurately segmenting the text region. Based on the binary image, the text region in the financial form is extracted, and a text detection result prediction box is generated.

[0047] For the first convolutional layer group of the residual network, the process of generating feature maps at the corresponding scale includes:

[0048] The financial form is input into the convolutional module of the first convolutional layer group to obtain the initial feature map;

[0049] The initial feature map is input into the channel attention module of the first convolutional layer group, and global max pooling and global average pooling are performed on the financial form respectively. The two pooled feature maps are then processed based on a shared multilayer perceptron to obtain a channel-weighted feature map.

[0050] The channel-weighted feature map is input into the spatial attention module of the first convolutional layer group. Global max pooling and global average pooling are performed on the channel-weighted feature map respectively and then concatenated. The concatenated feature map is then subjected to dilated convolution, and the feature map after dilated convolution is fused with the channel-weighted feature map to obtain the spatial weighted feature map.

[0051] The spatially weighted feature map is input into the residual module of the first convolutional layer group to generate a feature map of the corresponding scale.

[0052] The financial form image is input into the first convolutional layer group of the residual network. The initial feature map is obtained through convolution operations.

[0053] In one embodiment of the present invention, the process of generating feature maps of corresponding scales for each remaining convolutional layer group of the residual network includes:

[0054] The feature map generated by the previous convolutional layer group is input into the convolutional module of the current convolutional layer group to obtain the initial feature map of the scale corresponding to the current convolutional layer group;

[0055] The initial feature map is input into the residual module of the first convolutional layer group to generate a feature map of the corresponding scale.

[0056] like Figure 4 As shown, the first convolutional layer group includes a cascaded convolutional module, a channel attention module, a spatial attention module, and a residual module. The financial form is input into the convolutional module of the first convolutional layer group of the residual network (such as ResNet50). After initial convolution operations, an initial feature map F (with dimensions H*W*C) is obtained. The initial feature map F is then input into the channel attention module of the first convolutional layer group. Global average pooling and global max pooling are performed on the initial feature map F respectively, resulting in two feature vectors of size 1×1×C. These two feature vectors are then input into a shared multilayer perceptron (MLP), which consists of a hidden layer to reduce parameter computation. The activation size of the hidden layer is set to R / C = r×1×1, where R is the descent rate. The two feature vectors output by the MLP are added together and then processed through a sigmoid activation function to generate a channel attention map M. c (F). The processing procedure of the channel attention module is shown in formula (2):

[0057] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (2)

[0058] Where σ is the sigmoid function, AvgPool(F) represents global average pooling of the initial feature map F, MaxPool(F) represents global max pooling of the initial feature map F, and MLP is a multilayer perceptron. c (F) is the channel attention map generated by the multilayer perceptron. The channel attention map M... c (F) is multiplied element-wise with the initial input feature map F to obtain the channel-weighted feature map output by the channel attention module. In this way, important channel features can be focused on and unimportant channel features can be suppressed, thus providing a more powerful feature representation for the subsequent spatial attention module.

[0059] The channel-weighted feature map output from the channel attention module is input into the spatial attention module. First, global average pooling and global max pooling are performed again along the channel dimension on the channel-weighted feature map, resulting in two feature maps of size H×W×1. These two feature maps are then concatenated along the channel dimension to form a feature map of size H×W×2. To obtain stronger spatial location information and reduce the coarseness of text localization, the original conventional convolution in the spatial attention module is replaced with dilated convolution, and the concatenated feature map is then subjected to convolution. This invention considers the advantages of dilated convolution compared to conventional convolution: 1. Inserting blank regions between convolution kernels allows for the integration of information over a larger area, thus more effectively suppressing noise interference and improving the expressive power of spatial features. 2. Conventional convolution may lead to information loss when smoothing small regions, especially in edge and detail processing. 3. Dilated convolution helps retain more detailed features, thereby reducing the coarseness of text localization. Using dilated convolution can, to some extent, avoid the feature map size reduction caused by downsampling, thus maintaining high-resolution information. This is particularly important for pixel-level accuracy in localization tasks. The processing procedure of the spatial attention module is shown in formula (3):

[0060]

[0061] in, This is a dilated convolution operation with a 7×7 kernel, where σ is the sigmoid function, and AvgPool(M c (F) represents the channel-weighted feature map M. c (F) Perform global average pooling, MaxPool(M) c (F) represents the channel-weighted feature map M. c (F) Perform global max pooling, M s (F) is the spatial attention map. The generated spatial attention map M... s (F) Channel-weighted feature map M of the spatial attention module input c (F) Perform element-wise multiplication to obtain the spatially weighted feature map generated by the spatial attention module. Input the spatially weighted feature map into the residual module of the first convolutional layer group for further feature extraction and nonlinear transformation, outputting a feature map at the same scale as the first convolutional layer group. The residual module utilizes skip connections to alleviate the gradient vanishing problem in deep networks, ensuring effective feature propagation.

[0062] In one embodiment of the present invention, the process of generating feature maps of corresponding scales for each remaining convolutional layer group of the residual network includes:

[0063] The feature map generated by the previous convolutional layer group is input into the convolutional module of the current convolutional layer group to obtain the initial feature map of the scale corresponding to the current convolutional layer group;

[0064] The initial feature map is input into the residual module of the current convolutional layer group to generate a feature map of the corresponding scale.

[0065] For the remaining convolutional layers in the residual network (i.e., subsequent layers excluding the first one), the process of generating feature maps at the corresponding scale is as follows: The feature map generated by the previous convolutional layer is input into the convolutional module of the current convolutional layer, where further convolution operations are performed to extract deeper features, resulting in the initial feature map at the corresponding scale of the current convolutional layer. The initial feature map obtained from the current convolutional module is then input into the residual module of that convolutional layer. Through skip connections, the vanishing gradient phenomenon in deep networks is greatly alleviated. The resulting feature map at the corresponding scale retains important feature content.

[0066] S3. Extract text sequence features from the text region, and transcribe the extracted text sequence features based on an attention mechanism to extract the text sequence from the text region.

[0067] Text sequence features refer to the feature representations extracted from a text region, including but not limited to the shape, strokes, and arrangement of characters, which can identify specific text content. A text sequence refers to the readable text that is finally extracted after transcription processing.

[0068] Specifically, in one embodiment of the present invention, the step of extracting text sequence features from the text region and performing transcription processing on the extracted text sequence features based on an attention mechanism to extract the text sequence in the text region includes:

[0069] Features of the text region are extracted based on a convolutional neural network, and the extracted features are transformed according to contextual information using a bidirectional long short-term memory network to obtain a feature sequence.

[0070] Based on a bidirectional long short-term memory network and an attention mechanism, the feature sequence is transcribed into characters to extract the text sequence from the text region.

[0071] This invention converts text regions into text sequences using a character recognition algorithm model, which includes an encoder and a decoder. The encoder transforms the input image into a text feature sequence via a convolutional neural network and a bidirectional long short-term memory network. The decoder then generates the final text recognition result by transcribing the encoded text feature sequence using a character transcription method. This algorithm model employs a ResNet34 deep residual network as the backbone for feature extraction. To reduce the number of parameters, fully connected layers in the residual network are removed, retaining only convolutional and pooling layers. During feature extraction, a bidirectional long short-term memory network is used to process the text features and generate a feature sequence. Furthermore, the algorithm model incorporates an attention mechanism module and a bidirectional decoder structure to improve the accuracy of text recognition.

[0072] Specifically, such as Figure 5 As shown, the ResNet34 encoder is a convolutional recurrent neural network. First, the input text region image is fed into the ResNet34 convolutional neural network. Feature extraction is performed in the convolutional layers to obtain the convolutional feature map of the text region image. The feature map has a size of w×h×d, where w represents the width, h represents the height, and d represents the number of channels. Then, a pooling layer reduces the height h of the feature map to 1, resulting in a feature map of size w×1×d. The reduced feature map is then unfolded along its width to form a feature sequence f = (f1, f2, ..., f...) of length w. w ), where each f t The feature vector is used. A Bidirectional Long Short-Term Memory (BiLSTM) network is used to process the feature sequence, capturing the contextual information of the characters in the sequence to generate a more semantically meaningful feature sequence. Finally, the BiLSTM network transforms the feature map into a feature sequence. For each time step t in the feature sequence, the label prediction probability is calculated as shown in formula (4):

[0073] y t =softmax(Mf t +b),t=1,…,w (4)

[0074] Among them, y t The output at time step t is M, where M is the trained weight matrix, and f is the weight matrix. t Let b be the feature vector at time step t, b be the bias vector, and w be the length of the feature sequence.

[0075] The decoder is a BiLSTM based on bidirectional attention, capturing complementary semantic features from two directions. The decoder outputs a text sequence of length T, based on the hidden state s from the previous time step. t-1 The predicted value y at the previous time step t-1The decoder first predicts the output at the current time step t. t,i As shown in formula (5):

[0076] e t,i =w T tanh(Ws t-1 +Vh i +b) (5)

[0077] Among them, e t,i The decoder's hidden state s at the current time step t t-1 With encoder hidden state h i Attention score between, s t-1 h represents the hidden state of the decoder at time step t-1. i Let be the hidden state of the encoder at position i, where w, W, and V are weight parameters that need to be trained, and b is the bias. Based on the attention score e... t,i The attention weight α at the current time step t can be obtained. t.i As shown in formula (6):

[0078]

[0079] Where, α t.i Let α be the attention weight for encoder position i at time step t, and n be the length of the encoder output sequence. Attention weight α t.i This effectively represents the importance of each item in the encoder output. The hidden states of the encoder are weighted and summed using attention weights to obtain the context vector g at time step t. t As shown in formula (7):

[0080]

[0081] The context vector g t The hidden state s of the previous time step t-1 and the embedding vector f(y) of the predicted output from the previous time step t-1 As input, update the hidden state s of the decoder at the current time step t. t and output feature vector x t As shown in formula (8):

[0082] (x t ,s t =BiSLTM(s) t-1 ,[g t ,f(y t-1 (8)

[0083] Among them, [g t ,f(yt-1 f(y) represents concatenating the context vector and the embedding vector. t-1 ) represents the embedding vector of the predicted output at the previous time step. As shown in Equation (9), the output feature vector x of the decoder is used. t Calculate the output probability distribution p(y) at the current time step. t ):

[0084] p(y t = softmax(W0x) t +b0) (9)

[0085] Where W0 and b0 are the weight matrix and bias vector of the decoder output layer, respectively. Based on the output probability distribution, the class with the highest probability is selected to determine the predicted output y at the current time step t. t As shown in formula (10):

[0086] y t =argmaxp(y t (10)

[0087] It should be noted that the convergence of the loss function plays a decisive role in the final text recognition model structure. By selecting the loss function of a bidirectional decoder to constrain the final model, the running speed of the model and the convergence efficiency of the network structure are improved, thereby improving the text recognition performance of the model.

[0088] In the prediction process, the bidirectional decoder of this invention employs a beam search method to obtain the final decoding result from two directions, and then selects the result with the higher score as the output. The loss function L is shown in formula (11):

[0089]

[0090] Where p l p represents the probability density of the decoder from left to right. r This represents the probability density of the decoder from right to left. After training, during the model's inference phase, the beam search method is still used. Based on the decoding results corresponding to the two directions of the decoder, the one with the higher score is selected as the final decoding result.

[0091] S4. Encode and semantically enhance the text sequence to obtain a comprehensive vector sequence, and decode the comprehensive vector sequence to identify the named entities in the financial form.

[0092] Specifically, in one embodiment of the present invention, the step of encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify named entities in the financial form, includes:

[0093] The text sequence is input into a pre-trained language model to obtain an embedding vector sequence;

[0094] Based on a bidirectional long short-term memory network, the contextual features of the text sequence are captured, and the contextual features are encoded to obtain a context vector sequence.

[0095] Based on cosine similarity, similar vector combinations for each context vector in the context vector sequence are selected from the word vector library;

[0096] Based on the attention mechanism, the similar vectors in the similar vector combination are weighted and processed to obtain the semantic enhancement vector of each context vector, thus forming the semantic enhancement vector sequence of the context vector sequence;

[0097] The semantic enhancement vector sequence, the context vector sequence, and the embedding vector sequence are concatenated to obtain a comprehensive vector sequence.

[0098] Based on a multi-layer entity recognizer, the comprehensive vector sequence is decoded to identify and extract the nested named entities in the financial form; wherein, each layer of the entity recognizer includes a bidirectional long short-term memory network and a dual-kernel convolutional neural network.

[0099] like Figure 6 As shown, this paper addresses the identification of numerous nested entities with complex relationships and structural characteristics found in a financial corpus, aiming to derive nested entities from text sequences. First, the input identified text sequence is converted into a distributed vector representation, resulting in an embedding vector sequence. Then, a bidirectional long short-term memory (BiLSTM) network is used to extract text features, yielding a context vector sequence corresponding to the embedding vector sequence. Next, based on cosine similarity, several similar vectors for each context vector are found from a pre-trained word vector library (such as Tencentembedding). An attention mechanism is added to give more attention to the more important similar vectors in the current context, resulting in semantically enhanced vectors. The weights of the embedding vectors and semantically enhanced vectors are then balanced to form the final semantically enhanced vector sequence. Finally, the semantically enhanced vector sequence, the context vector sequence, and the embedding vector sequence from the pre-trained language model are concatenated to obtain a comprehensive vector sequence. This comprehensive vector sequence is then input into a multi-layer entity recognizer for decoding. Each layer of the entity recognizer includes a BiLSTM network and a dual-kernel convolutional neural network (CNN), progressively recognizing entities of different lengths. Ultimately, a set of financial named entities is obtained.

[0100] Specifically, the text sequence C = {c1, c2, c3, ..., c iThe input is fed into a pre-trained language model (such as BERT), where c i This represents the i-th character of the original input text sequence. Based on character-level word embedding, the pre-trained vector corresponding to each character is found from a pre-trained word vector library (such as TencentEmbedding), forming an embedding vector sequence V = {v1, v2, v3, ... v}. i For characters for which no corresponding pre-trained vector is found, a random vector will be initialized. Where v i Represents the i-th character c i The corresponding embedding vectors. The embedding vector sequence V is input into the encoding layer (i.e., a bidirectional long short-term memory network BiLSTM) to learn the contextual features of the text. BiLSTM is then used to extract text features, resulting in the context vector sequence. As shown in formula (12):

[0101]

[0102] Where BiLSTM is the encoding layer and Embed is the embedding layer. Let C be the context vector sequence and C be the text sequence. This invention does not perform decoding directly after encoding, but rather uses a semantic enhancement algorithm to semantically enhance the context vector sequence between character encoding and decoding. For each context vector... Based on cosine similarity, the n closest vectors are found from the pre-trained word vector library to form a similar vector combination V of the i-th context vector. i As shown in formula (13):

[0103] V i ={v i1 ,v i2 ,v i3 ...v ij ...v in} (13)

[0104] Among them, v ij For v i A similar vector. Using an attention mechanism, the weights of the similar vectors are calculated to highlight important similar vectors, thus completing the data augmentation process. Therefore, for each similar word, their weights can be calculated using formula (14):

[0105]

[0106] in, v is the latent vector (i.e., the context vector) of the input sequence after context encoding by the encoding layer. i,jLet x be the similarity vector of the i-th context vector. Then, the semantic enhancement vector x after weighting the i-th embedding vector is calculated using formula (15). i :

[0107]

[0108] Considering that similar vectors and original character vectors (i.e., embedding vectors) will have different contributions in different contexts, it is necessary to control the flow of information through a reset gate and measure their contribution, as shown in formula (16):

[0109] g=σ(R1·h i +R2·x i +b) (16)

[0110] Where R1 and R2 are trainable matrices, b is the corresponding bias, σ is the sigmoid function, and g is the reset gate. Then, the original context vector and the corresponding semantic enhancement vector are fused using formula (17) to obtain the fused vector:

[0111]

[0112] Among them, o i This is the final semantic enhancement vector obtained after fusion (i.e., the final semantic enhancement vector of the i-th embedding vector). Let i be the context vector of the i-th embedding vector. This is element-wise multiplication. For vector concatenation, [·] indicates that the result is a vector. The semantic enhancement vectors are arranged in sequence to obtain the semantic enhancement vector sequence o.

[0113] Furthermore, to improve model performance, this invention uses the embedding vector V = {v1, v2, v3, ... v} of the pre-trained language model. i}, semantic enhancement vector o, and the sequence of context vectors output by BiLSTM The vectors are concatenated and then the dimensionality is reduced by a fully connected layer to obtain the comprehensive vector sequence shown in formula (18).

[0114]

[0115] Here, Linear is a fully connected layer. Then, a decoder is used to synthesize the above vector sequence. Decoding is performed. The decoder consists of L layers of entity recognizers, each layer of which consists of a BiLSTM and a dual-core CNN, enabling the recognition of all entities with an entity span of [1,L]. The CNN aggregates hidden states at different levels, allowing the decoding layer to capture low-level information to recognize short entities and high-level information to recognize long entities, thus helping to alleviate error propagation problems and improve the accuracy of named entity recognition. The input of the first layer of the decoding layer is different from that of the other layers, as it has not undergone CNN convolution. To avoid this input difference affecting the model, the embedding is normalized before inputting into the LSTM, as shown in formulas (19) and (20):

[0116]

[0117]

[0118] in, The input is the l-th layer, BiLSTM is the decoding layer, LayerNorm is the layer normalization, and h is the input. i Let be the hidden state at position i in the l-th layer of the decoding layer. The final output yields a set of entity labels ε, as shown in formula (20):

[0119] ε={(E0,c0),(E1,c1),...,(E3,c n )} (20)

[0120] Where C = {c0, c1, ..., c n Let E = {E0, E1, ..., E} be the extracted set of named entities. n} represents the semantic label (i.e., the weight of the entity content) corresponding to the named entity. Through the above process, nested financial named entities can be identified and extracted from the text sequence to meet the needs of extracting complex financial text information.

[0121] In one embodiment of the present invention, the named entity includes a semantic tag and entity content. After encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify the named entity in the financial form, the method further includes:

[0122] Extract the semantic tag features and content features corresponding to each named entity;

[0123] The semantic tag features and entity content features of each named entity are combined to obtain the entity pair features of each named entity.

[0124] The entity pair features of each named entity are input into the feedforward neural network to obtain the question entity features and the answer entity features.

[0125] The question entity features and answer entity features of each named entity are input into a dual affine classifier to obtain the relationship score between each named entity.

[0126] For each named entity: select the other named entity with the highest relation score as the relation named entity of the current named entity, forming a relation entity pair.

[0127] like Figure 7 As shown, for judgment question pairs that only appear in financial forms, a pipelined approach is used for relation extraction, based on the named entity set ε output by the financial nested named entity recognition module. By processing the input entity set ε containing semantic tags and entity content, the relation extraction model is used to extract and match entity relations between questions and their answers in the text, ultimately obtaining a set of matched question pairs. Specifically, the relationships between entities are extracted from named entities using a relation extraction model, which includes at least a feedforward neural network and a biaffine classifier. When training the relation extraction model, for the already obtained entity set ∈, R = {r0, r1, ..., r...} is defined. m} represents a set of semantic relation tags between named entities. This is used to implement the relation extraction function F. RE :(ε,R)→ρ, i.e., the relation extraction function F RE Receive the named entity set ε and the relation label set R, and output the entity relation set ρ. ρ is shown in formula (21):

[0128] ρ={(question0,answer0,r0),...,(question n ,answer n ,r n )} (twenty one)

[0129] in, answer i These represent two named entities corresponding to the question and answer in the document, respectively. This method completes the extraction and matching of relationships between named entities and their corresponding pairs. During the model training phase, when constructing the dataset, a set of semantic relationship labels for possible relationships is manually built, establishing a candidate set of relationships for all semantic entities. This candidate set R contains all possible relationships between entities, such as "question-answer," "definition-term," etc.

[0130] Whether in the model training or inference phase, for each pair of entities obtained through the relation extraction function mentioned above... Semantic label features are obtained through a specific type of embedding layer, with each entity's semantic label corresponding to an embedding vector. The entity is paired with the feature vector x. iInput the two layers of feedforward neural network (FFNN) respectively, i.e., FFNN S and FFNN E The feature extraction of the question and answer entities is completed, as shown in formulas (22) and (23):

[0131]

[0132] Among them, h s (i), h e (i) are the entity features of the question and the entity features of the answer, respectively. It is the feature vector of the problem entity. These are the feature vectors of the answer entity. During relation extraction, these two feature vectors are used by a biaffine classifier to calculate the relationship score r between the question entity and the answer entity. m (i), as shown in formula (24):

[0133]

[0134] Among them, U m It is a biaffine transformation matrix, W m and b m These are the weight matrix and bias vector of the affine transformation, respectively. This involves concatenating vectors. By calculating relation scores, the type of relationship between each question-answer entity pair can be predicted, thus completing the relation extraction task.

[0135] like Figure 2 As shown, for a handwritten or printed financial form, the text sequence is first identified, and then named entity recognition is used to obtain the named entity set ∈: {(Entity category E1: Applicant's name, Entity content C1: Zhang San), (Entity category E2: ID number, Entity content C2: 11234177001251112)}, etc. Relation extraction reveals that the set of relational entity pairs is: (Applicant's name, Zhang San) and (ID number, 11234177001251112) have an identity association relationship.

[0136] Please see Figure 8The financial form recognition system 100 includes: a form acquisition module 110, used to acquire the financial form to be recognized; a text detection module 120, used to extract feature maps of different scales from the financial form according to a segmentation-based text detection algorithm and perform fusion processing, and identify and extract the text region in the financial form from the fused feature map; a text recognition module 130, used to extract the text sequence features in the text region, and perform transcription processing on the extracted text sequence features based on an attention mechanism to extract the text sequence in the text region; and a financial nested named entity recognition module 140, used to encode and semantically enhance the text sequence to obtain a comprehensive vector sequence, and decode the comprehensive vector sequence to identify the named entities in the financial form.

[0137] Specific limitations regarding the financial form recognition system can be found in the limitations of the financial form recognition method described above, and will not be repeated here. Each module in the aforementioned financial form recognition system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware format or independently of it, or stored in the memory of a computer device in software format, so that the processor can call the corresponding operations of each module.

[0138] It should be noted that, in order to highlight the innovative aspects of this invention, this embodiment does not include modules that are not closely related to solving the technical problems proposed by this invention, but this does not mean that there are no other modules in this embodiment.

[0139] Please see Figure 9 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a financial form recognition program.

[0140] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 12 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 1. Furthermore, the memory 12 can include both internal and external storage units of the electronic device 1. The memory 12 can be used not only to store application software and various types of data installed on the electronic device 1, such as identification codes for financial forms, but also to temporarily store data that has been output or will be output.

[0141] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules (such as financial form recognition programs) stored in the memory 12, and calls data stored in the memory 12 to perform various functions and process data of the electronic device 1.

[0142] The processor 13 executes the operating system of the electronic device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-described financial form recognition method.

[0143] For example, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a form acquisition module 110, a text detection module 120, a text recognition module 130, and a financial nested named body recognition module 140.

[0144] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium, which can be non-volatile or volatile. The software functional module stored in the storage medium includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute some functions of the financial form recognition method described in the various embodiments of this application.

[0145] In summary, the financial form recognition method, system, device, and medium disclosed in this invention, after acquiring the financial form to be recognized, utilizes a segmentation-based text detection algorithm to extract and fuse multi-scale feature maps, improving the detection accuracy for text regions of different scales and complex backgrounds. Text sequence features of the text regions are extracted and transcribed using an attention mechanism, enhancing the accuracy of text content recognition. The text sequence is encoded and semantically enhanced to obtain a comprehensive vector sequence, which is then decoded to successfully identify named entities in the financial form. This invention achieves automated, refined extraction and structured processing of financial form information. For images of complex financial forms, text recognition and extraction, as well as extraction of key named entities and their relationships, are performed to form a unified structured output. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0146] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method of recognizing a financial form, characterized by, The method comprises: acquiring a financial form to be recognized; extracting feature maps of different scales from the financial form and performing fusion processing according to a segmentation-based text detection algorithm, and recognizing and extracting a text region in the financial form from the fused feature maps; extracting text sequence features in the text region, and performing transcription processing on the extracted text sequence features based on an attention mechanism to extract a text sequence in the text region; encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to recognize a named entity in the financial form; the step of extracting feature maps of different scales from the financial form and performing fusion processing according to a segmentation-based text detection algorithm, and recognizing and extracting a text region in the financial form from the fused feature maps comprises: inputting the financial form into a residual network to perform feature processing through each convolutional layer group to generate feature maps of different scales; wherein the residual network comprises a plurality of cascaded convolutional layer groups, and each convolutional layer group generates a feature map of a scale; performing feature enhancement on the feature map of the smallest scale, and arranging the enhanced feature map of the smallest scale and the remaining feature maps of different scales in descending order of scale; based on a feature pyramid algorithm, fusing the feature maps according to their arrangement order to obtain a fused feature map; based on a differential binary text detection algorithm, recognizing and extracting a text region in the financial form from the fused feature map; the step of encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to recognize a named entity in the financial form comprises: inputting the text sequence into a pre-trained language model to obtain an embedded vector sequence; based on a bidirectional long short-term memory network, capturing context features of the text sequence, encoding the context features to obtain a context vector sequence; based on cosine similarity, screening similar vector combinations for each context vector in the context vector sequence from a word vector library; based on an attention mechanism, performing weighted processing on similar vectors in the similar vector combinations to obtain semantic enhancement vectors corresponding to each context vector, forming a semantic enhancement vector sequence of the context vector sequence; splicing the semantic enhancement vector sequence, the context vector sequence and the embedded vector sequence to obtain a comprehensive vector sequence; based on a multi-layer entity recognizer, decoding the comprehensive vector sequence to recognize and extract named entities nested in the financial form; wherein each layer of the entity recognizer comprises a bidirectional long short-term memory network and a double-kernel convolutional neural network.

2. The method of claim 1, wherein The convolution at the preset position in the residual network is deformable convolution.

3. The method of claim 1, wherein For the first convolutional layer group of the residual network, the process of generating a feature map of a corresponding scale comprises: inputting the financial form into a convolution module of the first convolutional layer group to obtain an initial feature map; The initial feature map is input into the channel attention module of the first convolutional layer group, and global max pooling and global average pooling are performed on the financial form respectively. The two pooled feature maps are then processed based on a shared multilayer perceptron to obtain a channel-weighted feature map. The channel-weighted feature map is input into the spatial attention module of the first convolutional layer group. Global max pooling and global average pooling are performed on the channel-weighted feature map respectively and then concatenated. The concatenated feature map is then subjected to dilated convolution, and the feature map after dilated convolution is fused with the channel-weighted feature map to obtain the spatial weighted feature map. The spatially weighted feature map is input into the residual module of the first convolutional layer group to generate a feature map of the corresponding scale.

4. The method of claim 1, wherein For each remaining convolutional layer group of the residual network, the process of generating feature maps at the corresponding scale includes: The feature map generated by the previous convolutional layer group is input into the convolutional module of the current convolutional layer group to obtain the initial feature map of the scale corresponding to the current convolutional layer group; The initial feature map is input into the residual module of the current convolutional layer group to generate a feature map of the corresponding scale.

5. The method of claim 1, wherein The process of extracting text sequence features from the text region and transcribing the extracted text sequence features based on an attention mechanism to extract the text sequence from the text region includes: Features of the text region are extracted based on a convolutional neural network, and the extracted features are transformed according to contextual information using a bidirectional long short-term memory network to obtain a feature sequence. Based on a bidirectional long short-term memory network and an attention mechanism, the feature sequence is transcribed into characters to extract the text sequence from the text region.

6. The method of claim 1, wherein The named entities include semantic tags and entity content. After encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and decoding the comprehensive vector sequence to identify the named entities in the financial form, the process further includes: Each named entity is input into a feedforward neural network for feature extraction, generating semantic label features and content features corresponding to each named entity; The semantic tag features and entity content features of each named entity are combined to obtain the entity pair features of each named entity. The entity pair features of each named entity are input into a dual affine classifier to obtain the relationship score between each named entity; For each named entity: select the other named entity with the highest relation score as the relation named entity of the current named entity, forming a relation entity pair.

7. A system for recognizing a financial form, characterized by The system includes: The form acquisition module is used to acquire financial forms to be identified. The text detection module is used to extract feature maps of different scales from the financial form according to a segmentation-based text detection algorithm and perform fusion processing, and to identify and extract the text regions in the financial form from the fused feature maps. The text recognition module is used to extract text sequence features from the text region and transcribe the extracted text sequence features based on an attention mechanism to extract the text sequence from the text region. The financial nested named entity recognition module is used to encode and semantically enhance the text sequence to obtain a comprehensive vector sequence, and to decode the comprehensive vector sequence to identify the named entities in the financial form; The step of extracting feature maps of different scales from the financial form using a segmentation-based text detection algorithm and fusing them, and then identifying and extracting text regions from the fused feature maps, includes: The financial form is input into a residual network, and feature processing is performed through each convolutional layer group to generate feature maps of different scales; wherein, the residual network includes multiple cascaded convolutional layer groups, and each convolutional layer group generates a feature map of one scale; Feature enhancement is performed on the feature map at the smallest scale, and the enhanced feature map at the smallest scale and the feature maps at the remaining scales are arranged in descending order of scale. Based on the feature pyramid algorithm, the feature maps are fused according to their arrangement order to obtain the fused feature map. Based on the differential binarization text detection algorithm, the text region in the financial form is identified and extracted from the fused feature map; The process of encoding and semantically enhancing the text sequence to obtain a comprehensive vector sequence, and then decoding the comprehensive vector sequence to identify named entities in the financial form, includes: The text sequence is input into a pre-trained language model to obtain an embedding vector sequence; Based on a bidirectional long short-term memory network, the contextual features of the text sequence are captured, and the contextual features are encoded to obtain a context vector sequence. Based on cosine similarity, similar vector combinations for each context vector in the context vector sequence are selected from the word vector library; Based on the attention mechanism, the similar vectors in the similar vector combination are weighted and processed to obtain the semantic enhancement vector of each context vector, thus forming the semantic enhancement vector sequence of the context vector sequence; The semantic enhancement vector sequence, the context vector sequence, and the embedding vector sequence are concatenated to obtain a comprehensive vector sequence. Based on a multi-layer entity recognizer, the comprehensive vector sequence is decoded to identify and extract the nested named entities in the financial form; wherein, each layer of the entity recognizer includes a bidirectional long short-term memory network and a dual-kernel convolutional neural network.

8. An electronic device, comprising: The electronic device includes: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to implement the financial form identification method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the financial form identification method according to any one of claims 1 to 6.

Images