Form content extraction method and system

Abstract

The invention provides a form content extraction method and system. The scheme comprises the following steps: acquiring a form image of which the content is to be extracted; extracting a text and text position information corresponding to the text based on the obtained form image; based on the extracted text and the text position information corresponding to the text, utilizing a pre-trained text label identification model to realize label identification of the text in the form; and based on the obtained text and the text label, realizing extraction of form contents.

Description

Technical Field

[0001] The present invention belongs to the technical field of document content extraction, and in particular relates to a form content extraction method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Traditional document content extraction usually relies on human intervention. SmartFIX is designed to process various files such as fixed-format tables and unstructured letters in any format. Schuster et al. used a rule-based filtering method to automatically extract company names from financial news. Template matching-based methods are widely used to extract one or more targets. However, these methods often require the pre-design of a large number of rules or the design of templates based on location coordinates, and the migration cost is relatively high.

[0004] Based on the above problems, Chiu et al. proposed an innovative hybrid model that combines BiLSTM and CNN to automatically detect and extract word and character-level features. It does not require tedious feature engineering and deep vocabulary knowledge, simplifies the process and meets high-performance requirements. Huang et al. proposed a BI-LSTM-CRF model for sequence labeling tasks. This model can combine BiLSTM and CRF (conditional random field) at the same time. It not only utilizes past and future input features, but also considers sentence-level label information, making it more powerful and flexible in processing natural language processing tasks. The Post-OCR parsing solution extracts text paragraphs and coordinates through OCR, serializes and performs BIO (biological marker) marking, and finally groups and merges to generate parsing results. Jiang et al. proposed to add text block coordinate embedding features directly to the BiLSTM-CRF model. However, these methods only rely on the text itself and its corresponding position information for content extraction. For content extraction of multimodal documents (such as forms, where there is a certain correlation between internal text data), such methods cannot effectively obtain the correlation between texts, resulting in incomplete extracted content information. Summary of the invention

[0005] The embodiment of the present invention provides a form content extraction method and system to solve the problem that the prior art only relies on the text itself and its corresponding position information to extract content, but ignores the association relationship between text entities in the form, resulting in incomplete extracted content information.

[0006] According to a first aspect of an embodiment of the present invention, a form content extraction method is provided, comprising:

[0007] Obtain the form image to be extracted;

[0008] Based on the obtained form image, extract the text and its corresponding text position information;

[0009] Based on the extracted text and its corresponding text position information, the pre-trained text label recognition model is used to realize label recognition of the text in the form;

[0010] The text tag recognition model specifically performs the following processing: based on the extracted text and its corresponding text position information, respectively construct text embedding representation, position information embedding representation and layout embedding representation; based on the text embedding representation, position information embedding representation and layout embedding representation, a multimodal feature representation is obtained by using a multimodal encoder introduced into the attention refinement module; based on the multimodal feature representation, a decoder introduced into the attention refinement module is used to obtain the recognition result of the text tag; wherein the attention refinement module is used to construct an initial similarity matrix by using the attention score of the input data obtained based on the attention mechanism; a refined matrix is ​​constructed based on the initial similarity matrix; based on the initial similarity matrix and the refined matrix, the content vector obtained by the attention mechanism is combined to obtain the output result of the attention refinement module;

[0011] Based on the obtained text and text tags, the form content is extracted.

[0012] Furthermore, the construction of the text embedding representation, the position information embedding representation and the layout embedding representation is specifically as follows: the extracted text is segmented and each word is converted into an index in a vocabulary, and the index is used as a one-dimensional position information embedding representation; the word vector of each segmented word in the text and the one-dimensional position information embedding representation are added to obtain the text embedding representation; based on the width, height and boundary coordinate values ​​of the text box corresponding to the text extraction, the layout embedding representation corresponding to the text is constructed.

[0013] Furthermore, the multimodal encoder that introduces the attention refinement module specifically includes a multi-head attention refinement module and a two-layer feedforward neural network, wherein the multi-head attention refinement module is composed of several attention refinement modules, and the text embedding representation, the position information embedding representation and the layout embedding are respectively used as the input of each attention refinement module in the multi-head attention refinement module, and the outputs of each attention refinement module are spliced ​​and linearly transformed to obtain splicing features; the splicing features are sequentially processed by residual connection, normalization and fully connected network to obtain multimodal feature representation.

[0014] Furthermore, the decoder that introduces the attention refinement module specifically includes a first decoder and a second decoder. Based on the obtained multimodal feature representation, the category label corresponding to the text is obtained through the first decoder. At the same time, based on the obtained category label, the relationship between the texts is obtained through the second decoder.

[0015] Furthermore, the attention refinement module is specifically expressed as follows:

[0016]

[0017] R=reshape(norm(max(0,conv(reshape(α t )))W R ))

[0018] Among them, A represents the attention score matrix, t∈[0,L), t represents the position, α t represents the attention score at position t, R represents the refined matrix after batch normalization and dimension adjustment, and W R ∈d c ×n is the trainable parameter matrix, d c is the intermediate dimension of the convolutional layer, and n represents the number of attention heads.

[0019] Furthermore, the text and its corresponding text position information are extracted by using an optical character recognition method.

[0020] According to a second aspect of an embodiment of the present invention, a form content extraction system is provided, comprising:

[0021] A data acquisition unit, which is used to acquire a form image to be extracted;

[0022] A basic information extraction unit, which is used to extract text and its corresponding text position information based on the obtained form image;

[0023] A text label recognition unit, which is used to realize label recognition of text in a form based on the extracted text and its corresponding text position information using a pre-trained text label recognition model; wherein the text label recognition model specifically performs the following processing procedures: based on the extracted text and its corresponding text position information, respectively construct a text embedding representation, a position information embedding representation and a layout embedding representation; based on the text embedding representation, the position information embedding representation and the layout embedding representation, a multimodal feature representation is obtained using a multimodal encoder introduced into an attention refinement module; based on the multimodal feature representation, a text label recognition result is obtained using a decoder introduced into an attention refinement module; wherein the attention refinement module is used to construct an initial similarity matrix using the attention score of the input data obtained based on the attention mechanism; a refined matrix is ​​constructed based on the initial similarity matrix; based on the initial similarity matrix and the refined matrix, the output result of the attention refinement module is obtained in combination with the content vector obtained by the attention mechanism;

[0024] The form content extraction unit is used to extract the form content based on the obtained text and text tags.

[0025] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored and running on the memory, wherein the processor implements the form content extraction method when executing the program.

[0026] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the form content extraction method is implemented.

[0027] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the form extraction method is implemented.

[0028] One or more of the above technical solutions have the following beneficial effects:

[0029] (1) The present invention provides a form content extraction method and system. In the form content extraction process, the scheme introduces an overall layout embedding representation of the form document, combines the text and its corresponding text position information, performs label recognition on the text in the form based on multimodal features, and obtains the association relationship between the text in the form through label recognition; and uses the combination of the text label and the text itself as the result of form content extraction, thereby effectively enriching the integrity of form content extraction.

[0030] (2) The present invention provides an attention refinement module, through which the attention score can be weighted and improved, so that text recognition can be performed more accurately.

[0031] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0033] Figure 1 This is an overall flow chart of a form extraction method described in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the form image text extraction and label recognition results described in an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of the local and overall structure of the attention refinement module described in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0038] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0039] like Figure 1 As shown, the embodiment of the present invention provides a form content extraction method, which specifically includes the following processing procedures:

[0040] Step 1: Obtain the form image to be extracted;

[0041] In a specific implementation, a digital camera may be used to capture the form image.

[0042] Step 2: Based on the obtained form image, extract the text and its corresponding text position information;

[0043] In a specific implementation, the text and its corresponding text position information are extracted by using an optical character recognition method, which specifically includes: inputting the form image into a DBNet++ network to obtain the position of all texts, and using POS m , m represents the serial number of the text position, the result is as follows Figure 2 As shown in (a); let IMG m Indicates POS m The corresponding image. m Input into the CRNN network to get POS m The text of the location is in TXT m Indicates that the result is Figure 2 As shown in (b) in .

[0044] Step 3: Based on the extracted text and its corresponding text position information, use the pre-trained text label recognition model to realize label recognition of the text in the form;

[0045] The text tag recognition model specifically performs the following processing: based on the extracted text and its corresponding text position information, respectively construct text embedding representation, position information embedding representation and layout embedding representation; based on the text embedding representation, position information embedding representation and layout embedding representation, a multimodal feature representation is obtained by using a multimodal encoder introduced into the attention refinement module; based on the multimodal feature representation, a decoder introduced into the attention refinement module is used to obtain the recognition result of the text tag; wherein the attention refinement module is used to construct an initial similarity matrix by using the attention score of the input data obtained based on the attention mechanism; a refined matrix is ​​constructed based on the initial similarity matrix; based on the initial similarity matrix and the refined matrix, the content vector obtained by the attention mechanism is combined to obtain the output result of the attention refinement module;

[0046] In a specific implementation, the construction of the text embedding representation, the position information embedding representation and the layout embedding representation is specifically as follows: the extracted text is segmented and each word is converted into an index in a vocabulary, and the index is used as a one-dimensional position information embedding representation; the word vector of each segmented word in the text and the one-dimensional position information embedding representation are added to obtain the text embedding representation; based on the width, height and boundary coordinate values ​​of the text box corresponding to the text extraction, the layout embedding representation corresponding to the text is constructed.

[0047] In a specific implementation, the multimodal encoder that introduces the attention refinement module specifically includes a multi-head attention refinement module and a two-layer feedforward neural network, wherein the multi-head attention refinement module is composed of several attention refinement modules, and the text embedding representation, the position information embedding representation and the layout embedding are respectively used as the input of each attention refinement module in the multi-head attention refinement module, and the outputs of each attention refinement module are spliced ​​and linearly transformed to obtain splicing features; the splicing features are sequentially processed by residual connection, normalization and fully connected network to obtain a multimodal feature representation.

[0048] In a specific implementation, the decoder that introduces the attention refinement module specifically includes a first decoder and a second decoder. Based on the obtained multimodal feature representation, the category label corresponding to the text is obtained through the first decoder. At the same time, based on the obtained category label, the relationship between the texts is obtained through the second decoder.

[0049] For ease of understanding, the text label recognition model in this embodiment is described in detail below with reference to the accompanying drawings:

[0050] The text tag recognition model includes a multimodal information embedding layer, an encoder that introduces an attention refinement module (the processing result is as follows Figure 2 (c)) and the decoder (processing results are shown in Figure 2 The processing results are shown in (d). Figure 2 As shown in (c), specifically:

[0051] Modal information embedding layer:

[0052] The modal information embedding layer consists of a trainable embedding matrix, each row of which corresponds to an embedding vector of an input element. By mapping the input data to the corresponding rows in the embedding matrix, the embedded representation of the input data can be obtained. The multimodal information embedding part includes text embedding, layout embedding, and one-dimensional position information embedding. In the text embedding part, the text in the test report OCR is first segmented, and each word is converted into an index in the vocabulary. These indexes can be regarded as one-dimensional position vectors, which are used to represent the position of the text in the sequence. Then, the word vector and the one-dimensional position vector are added to obtain the text vector of the input encoder representing the text. The i-th text embedding T i Defined as:

[0053] T i =TokenEmb(t i )+PosEmb1D(i),0≤i≤L (1)

[0054] Among them, t iRepresents the i-th word vector, L represents the sequence length, TokenEmb() represents the embedding representation of the word vector, and PosEmb1D() represents the embedding representation of the position information.

[0055] The layout vector is a vector representation of spatial layout information. The entire test sheet image is regarded as a coordinate system (with the upper left as the origin). For the coordinate range covered by each word in the image, a bounding box parallel to the coordinate axis is used to represent the 2D position information. The width and height coordinates of the bounding box are both identified by the width and height of the box in the OCR result. After the text is segmented, different tokens are obtained, and each token shares the text-level bounding box. This method normalizes all coordinates so that their values ​​are in [0,1000]. Then two vector layers are used to encode the position features of the x-axis and y-axis respectively. The layout vector layer describes each bounding box through four coordinate values ​​(defining its upper, lower, left, and right boundaries) and six key parameters, namely width and height. These parameters together determine the precise position of the bounding box on the two-dimensional plane. For example, in a test sheet, there is a text box called "average hemoglobin concentration". First, the height h and width w of this bounding box are identified. After segmentation, different tokens such as "average", "hemoglobin", and "concentration" are obtained. These three tokens share the entire text box. Then, four coordinate values ​​are used to encode the text box, and x0 (upper boundary), x1 (lower boundary), y0 (left boundary) and y1 (right boundary) are obtained, and normalized. Finally, the four parameters of the upper, lower, left and right boundaries, together with the height and width, determine the specific position of the text box. Subsequently, these six parameters are converted into corresponding vector representations and integrated into a comprehensive layout vector through splicing operations. For the i-th text boundary box i , the layout vector P in the input encoder i It is expressed as:

[0056] P i =Concat(PosEmb2D x (x0,x1,w),PosEmb2D y (y0,y1,h)),0≤i <L (2)

[0057] Among them, (x0, y0) represents the coordinates of the upper left corner, (x1, y1) represents the coordinates of the lower right corner, w represents the width of the bounding box, and h represents the height of the bounding box. x () and PosEmb2D y () is the two-dimensional embedding representation of the position coordinates.

[0058] Introducing the encoder of the attention refinement module:

[0059] The multimodal encoder with attention refinement is built on the basis of Transformer. Its core components include a multi-head attention refinement module and two layers of feedforward neural network. At the same time, both networks have added residual connections, and layer normalization is performed after the residual connection. This section focuses on the multi-head attention refinement module. The improved refined attention is as follows Figure 3 As shown in (a) in Fig. 3, the multi-head attention refinement module consists of multiple self-attention refinement layers. Each layer is created by adding a refinement module to the traditional attention mechanism.

[0060] The self-attention mechanism starts from the input X={x1,x2,...x L}∈L×d model First, we multiply three different matrices to get Q (query for matching other units), K (key for being matched by other units), and V (value of information to be extracted). The three vector formulas are:

[0061] Q=XW Q ,K=XW K ,V=XW V (3)

[0062] Among them, W Q , W K ∈d model ×d k , W V ∈d model ×d v is the trainable parameter matrix, L is the length of the input sequence, and d model is the model depth, d k is the depth of each attention head.

[0063] Then, the attention score A is calculated as follows:

[0064]

[0065] Among them, d k The setting is to avoid gradient disappearance, and α is the similarity matrix between input units.

[0066] This embodiment proposes an attention refinement module to weight and improve the attention score, so that text can be recognized more accurately. Let t∈[0,L), t represents the position, α t represents the attention score at position t, R represents the refined matrix after batch normalization and dimension adjustment, then

[0067]

[0068] and

[0069] R=reshape(norm(max(0,conv(reshape(α t )))W R )) (6)

[0070] Among them, the convolution kernel size of the convolution layer is 5, W R ∈d c ×n is the trainable parameter matrix, d c is the middle dimension of the convolutional layer. n represents the number of attention heads. Finally, the final similarity matrix B is obtained by subtracting the fine term R from the originally generated similarity matrix α. The calculation formula is:

[0071] B=α-R (7)

[0072] After B is softmaxed, it is weighted with V to get the result of an attention head. The i-th attention head H i It can be expressed as:

[0073] H i =softmax(B)V i (8)

[0074] The final output of the multi-head attention refinement module is to concatenate n attention heads and then perform a linear transformation, that is,

[0075] H=concat(H1,H2,...,H n )W n (9)

[0076] Where W n ∈nd k ×d model is the trainable parameter matrix.

[0077] The output features of the multi-head attention refinement module are input into the two-layer fully connected network after residual connection and normalization operation to further extract features and enhance the model's expression ability. The specific calculation process is:

[0078] FFN(x)=max(0,xW1+b1)W2+b2 (10)

[0079] Among them, the dimension of input x is d model , the output dimension of the second linear layer is equal to the dimension of x, W1 and W2 represent the weight parameters of the first and second layers respectively, and b1 and b2 represent the bias parameters of the first and second layers respectively.

[0080] Stacking N layers of encoding layers consisting of a multi-head attention refinement module and two layers of feedforward neural networks constitutes a multimodal encoder.

[0081] Decoder with attention refinement module:

[0082] The task of the decoder part is to generate a specific output based on the multimodal information generated by the encoder. The decoder includes two sub-decoders, which handle the SER task and the RE task respectively. The SER task is essentially a classification task. Its main purpose is to assign a predefined entity category label to the input sequence at each time step, that is, the model is required to classify entities such as "Header", "Result", and "Reference" in a given test order image. The SER decoder part first uses a fully connected layer to classify the output results of the encoder. The predefined test order image contains 11 types of entities, and each category is represented by BIO annotation. The prediction result is then combined with the OCR result. By counting the predicted labels of all text segments of each text, the result with the most predictions in all text segments is selected as the label of the text. The model performance is evaluated by calculating the cross entropy loss between each predicted entity category and the true annotation to assist network training. The RE task is to extract the relationship between entities from the predicted entity category. In this embodiment, the RE task needs to establish all possible relationship pairs for all entities from a given test order image, and send all possible relationship results to the Biaffine Attention classifier. Valid relationship pairs are predicted as "1", and vice versa as "0". The test order image is predicted by the RE decoder to produce the key-value representation we pre-defined, such as the ordering department and clinical diagnosis results.

[0083] Step 4: Extract the form content based on the obtained text and text tags.

[0084] In this way, during the form content extraction process, the above scheme introduces the overall layout embedding representation of the form document, combines the text and its corresponding text position information, performs label recognition on the text in the form based on multimodal features, and obtains the association relationship between the text in the form through label recognition; and uses the combination of the text label and the text itself as the result of form content extraction, which effectively enriches the integrity of form content extraction.

[0085] In one or more implementations, an embodiment of the present invention corresponds to the above-mentioned form content extraction method and provides a form content extraction system, including:

[0086] A data acquisition unit, which is used to acquire a form image to be extracted;

[0087] A basic information extraction unit, which is used to extract text and its corresponding text position information based on the obtained form image;

[0088] A text label recognition unit, which is used to realize label recognition of text in a form based on the extracted text and its corresponding text position information using a pre-trained text label recognition model; wherein the text label recognition model specifically performs the following processing procedures: based on the extracted text and its corresponding text position information, respectively construct a text embedding representation, a position information embedding representation and a layout embedding representation; based on the text embedding representation, the position information embedding representation and the layout embedding representation, a multimodal feature representation is obtained using a multimodal encoder introduced into an attention refinement module; based on the multimodal feature representation, a text label recognition result is obtained using a decoder introduced into an attention refinement module; wherein the attention refinement module is used to construct an initial similarity matrix using the attention score of the input data obtained based on the attention mechanism; a refined matrix is ​​constructed based on the initial similarity matrix; based on the initial similarity matrix and the refined matrix, the output result of the attention refinement module is obtained in combination with the content vector obtained by the attention mechanism;

[0089] The form content extraction unit is used to extract the form content based on the obtained text and text tags.

[0090] It can be understood that the system described in this embodiment corresponds to the method described in the above embodiment, and its technical details have been described in detail in Embodiment 1 and will not be repeated here.

[0091] In further embodiments, there is also provided:

[0092] An electronic device includes a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the above embodiment. For the sake of brevity, no further description is given here.

[0093] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0094] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0095] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in the above embodiment is completed.

[0096] The method in the above embodiment can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0097] A computer program product, comprising a computer program, wherein the computer program implements the form content extraction method when executed by a processor.

[0098] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in the present embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A form content extraction method and system, characterized in that: include: Obtain the form image to be extracted; Based on the obtained form image, extract the text and its corresponding text position information; Based on the extracted text and its corresponding text position information, the pre-trained text label recognition model is used to realize label recognition of the text in the form; The text tag recognition model specifically performs the following processing: based on the extracted text and its corresponding text position information, respectively construct text embedding representation, position information embedding representation and layout embedding representation; based on the text embedding representation, position information embedding representation and layout embedding representation, a multimodal feature representation is obtained by using a multimodal encoder introduced into the attention refinement module; based on the multimodal feature representation, a decoder introduced into the attention refinement module is used to obtain the recognition result of the text tag; wherein the attention refinement module is used to construct an initial similarity matrix by using the attention score of the input data obtained based on the attention mechanism; a refined matrix is ​​constructed based on the initial similarity matrix; based on the initial similarity matrix and the refined matrix, the content vector obtained by the attention mechanism is combined to obtain the output result of the attention refinement module; Based on the obtained text and text tags, the form content is extracted.

2. A form content extraction method and system as claimed in claim 1, characterized in that: The construction of the text embedding representation, the position information embedding representation and the layout embedding representation is specifically as follows: the extracted text is segmented and each word is converted into an index in a vocabulary, and the index is used as a one-dimensional position information embedding representation; the word vector of each segmented word in the text and the one-dimensional position information embedding representation are added to obtain the text embedding representation; based on the width, height and boundary coordinate values ​​of the text box corresponding to the text extraction, the layout embedding representation corresponding to the text is constructed.

3. A form content extraction method and system as claimed in claim 1, characterized in that: The multimodal encoder that introduces the attention refinement module specifically includes a multi-head attention refinement module and a two-layer feedforward neural network, wherein the multi-head attention refinement module is composed of a plurality of attention refinement modules, and the text embedding representation, the position information embedding representation and the layout embedding are respectively used as the input of each attention refinement module in the multi-head attention refinement module, and the outputs of each attention refinement module are spliced ​​and linearly transformed to obtain splicing features; the splicing features are sequentially processed by residual connection, normalization and fully connected network to obtain a multimodal feature representation.

4. A form content extraction method and system as claimed in claim 1, characterized in that: The decoder that introduces the attention refinement module specifically includes a first decoder and a second decoder. Based on the obtained multimodal feature representation, the category label corresponding to the text is obtained through the first decoder. At the same time, based on the obtained category label, the relationship between the texts is obtained through the second decoder.

5. A form content extraction method and system as claimed in claim 3, characterized in that: The attention refinement module is specifically expressed as follows: R=reshape(norm(max(0,conv(reshape(α t )))W R )) Among them, A represents the attention score matrix, t∈[0,L), t represents the position, α t represents the attention score at position t, R represents the refined matrix after batch normalization and dimension adjustment, and W R ∈d c ×n is the trainable parameter matrix, d c is the intermediate dimension of the convolutional layer, and n represents the number of attention heads.

6. A form content extraction method and system as claimed in claim 1, characterized in that: The text and its corresponding text position information are extracted by using an optical character recognition method.

7. A form content extraction system, characterized in that: include: A data acquisition unit, which is used to acquire a form image to be extracted; A basic information extraction unit, which is used to extract text and its corresponding text position information based on the obtained form image; A text label recognition unit, which is used to realize label recognition of text in a form based on the extracted text and its corresponding text position information using a pre-trained text label recognition model; wherein the text label recognition model specifically performs the following processing procedures: based on the extracted text and its corresponding text position information, respectively construct a text embedding representation, a position information embedding representation and a layout embedding representation; based on the text embedding representation, the position information embedding representation and the layout embedding representation, a multimodal feature representation is obtained using a multimodal encoder introduced into an attention refinement module; based on the multimodal feature representation, a text label recognition result is obtained using a decoder introduced into an attention refinement module; wherein the attention refinement module is used to construct an initial similarity matrix using the attention score of the input data obtained based on the attention mechanism; a refined matrix is ​​constructed based on the initial similarity matrix; based on the initial similarity matrix and the refined matrix, the output result of the attention refinement module is obtained in combination with the content vector obtained by the attention mechanism; The form content extraction unit is used to extract the form content based on the obtained text and text tags.

8. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, the form content extraction method as described in any one of claims 1-6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the form content extraction method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the form extraction method according to any one of claims 1 to 6 is implemented.

Images