Document identification method and training method and device of document identification model

Abstract

The invention provides a document recognition method and a training method and device of a document recognition model, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning and big data. The document identification method comprises the steps of performing feature extraction of each modal on multi-modal data of a document page in a document to obtain a plurality of modal features; determining a first fusion feature of the document page according to respective attention features of the plurality of modal features; and determining an identification result of the document according to the first fusion feature of the document page. The invention further provides electronic equipment, a computer readable storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and big data technology, and specifically to a document recognition method, a training method and a device for a document recognition model. Background Art

[0002] With the continuous development of artificial intelligence technology, deep learning technology has also been applied in many fields. For example, deep learning models are used to assist in online document recognition and document recommendation.

[0003] However, the number of documents in the document library is huge and the quality is uneven, so the document recognition accuracy is not high. Summary of the invention

[0004] The present disclosure provides a training method and device based on a document recognition method and a document recognition model.

[0005] According to one aspect of the present disclosure, a document recognition method is provided, comprising: extracting features of each modality of multimodal data of a document page in a document to obtain multiple modal features; determining a first fusion feature of the document page based on attention features of each of the multiple modal features; and determining a recognition result of the document based on the first fusion feature of the document page.

[0006] According to another aspect of the present disclosure, a method for training a document recognition model is provided, comprising: obtaining a training sample set, wherein the training sample set includes a plurality of sample documents; training a document recognition model using the sample documents in the training sample set, so that the document recognition model performs the following operations: extracting features of each modality of multimodal data of a document page in the sample document to obtain a plurality of modal features; determining a first fusion feature of the document page based on respective attention features of the plurality of modal features; and determining a recognition result of the sample document based on the first fusion feature of the document page.

[0007] According to another aspect of the present disclosure, a document recognition device is provided, including: a feature extraction module, used to extract features of each modality of multimodal data of a document page in a document to obtain multiple modal features; a fusion module, used to determine a first fusion feature of the document page based on the attention features of each of the multiple modal features; and a recognition module, used to determine a recognition result of the document based on the first fusion feature of the document page.

[0008] According to another aspect of the present disclosure, a training device for a document recognition model is provided, comprising: an acquisition module, used to acquire a training sample set, wherein the training sample set includes multiple sample documents; a training module, used to train a document recognition model using the sample documents in the training sample set, so that the document recognition model performs the following operations: extracting features of each modality of multimodal data of a document page in the sample document to obtain multiple modal features; determining a first fusion feature of the document page based on the attention features of each of the multiple modal features; and determining a recognition result of the sample document based on the first fusion feature of the document page.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 An exemplary system architecture to which a document recognition method, a training method and a device for a document recognition model according to an embodiment of the present disclosure can be applied is schematically shown;

[0015] Figure 2 A flowchart of a document recognition method according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 3 A diagram schematically illustrates an application scenario of determining a recognition result of a document according to an embodiment of the present disclosure;

[0017] Figure 4A A schematic diagram of a scenario for determining a first fusion feature according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 4BA schematic diagram of a scene for determining a first fusion feature according to another embodiment of the present disclosure is schematically shown;

[0019] Figure 4C A schematic diagram of a scenario for determining a first fusion feature according to yet another embodiment of the present disclosure is schematically shown;

[0020] Figure 5 Schematically illustrates an exemplary system architecture of a feature extraction layer and a feature fusion layer of a document recognition model according to an embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram schematically shows a scenario of determining a recognition result based on multiple document pages according to an embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram schematically shows a scenario of a method for training a document recognition model according to an embodiment of the present disclosure;

[0023] Figure 8 A block diagram of a document recognition device according to an embodiment of the present disclosure is schematically shown;

[0024] Fig. 9 A block diagram schematically shows a training device for a document recognition model according to an embodiment of the present disclosure; and

[0025] Fig.10 A block diagram of an electronic device suitable for implementing a document recognition method and a training method for a document recognition model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] At present, the quality of hundreds of millions of documents in different fields in the document library is uneven. The document quality management of the document library and the optimization of the document search experience all rely on the identification of high-quality documents. For example, in the search scenario, screening out high-quality documents and pushing them to users in a targeted manner is a key factor in improving CTR (Click-Through Rate) and CVR (Conversion Rate).

[0028] Document recognition is essentially a multimodal content understanding problem. The model can extract features from the multimodal content information (vision, text) of a document and determine whether the document is high-quality. To address the problem of multimodal content understanding, the following solutions are currently available: 1) Feature extraction based on a pre-trained model of vision-text. However, this solution currently only supports text input of less than 100 characters. However, documents, especially high-quality documents, are generally long texts with rich content. The input character limit of this solution will cause a lot of information loss, affecting document recognition performance. 2) Feature extraction based on vision and text models respectively, and then multimodal features are fused. However, the multimodal fusion method under this solution is relatively simple and does not fully couple the visual and text features, which affects document recognition performance.

[0029] Therefore, in order to at least partially solve the above technical problems, the embodiments of the present disclosure provide a document recognition method, including: performing feature extraction of each modality on multimodal data of a document page in a document to obtain multiple modal features; determining a first fusion feature of the document page based on the attention features of each of the multiple modal features; and determining a recognition result of the document based on the first fusion feature of the document page.

[0030] Figure 1 An exemplary system architecture to which a document recognition method, a training method and an apparatus for a document recognition model according to an embodiment of the present disclosure can be applied is schematically shown.

[0031] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the document recognition method, document recognition model training method and apparatus can be applied may include a terminal device, but the terminal device may implement the method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0037] It should be noted that the document recognition method and the document recognition model training method provided in the embodiments of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Correspondingly, the document recognition device and the document recognition model training device provided in the embodiments of the present disclosure can also be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0038] Alternatively, the document recognition method and the document recognition model training method provided in the embodiments of the present disclosure may also be generally executed by the server 105. Accordingly, the document recognition device and the document recognition model training device provided in the embodiments of the present disclosure may generally be arranged in the server 105. The document recognition method and the document recognition model training method provided in the embodiments of the present disclosure may also be executed by a server or a server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the document recognition device and the document recognition model training device provided in the embodiments of the present disclosure may also be arranged in a server or a server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] For example, when a user uploads a document through the first terminal device 101, the second terminal device 102, and the third terminal device 103, or uses the document recognition function, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can extract features of each modality of the multimodal data of the document page in the document to obtain multiple modal features; determine the first fusion feature of the document page based on the attention features of each of the multiple modal features; and determine the recognition result of the document based on the first fusion feature of the document page.

[0040] Alternatively, the user can also send documents to the server 105 through the first terminal device 101, the second terminal device 102, and the third terminal device 103, and use the server 105 to perform the above-mentioned document recognition method to determine and return the recognition results to the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0043] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0044] Figure 2 The flowchart of the document recognition method according to the embodiment of the present disclosure is schematically shown.

[0045] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0046] In operation S210, feature extraction of each modality is performed on multimodal data of a document page in a document to obtain a plurality of modality features.

[0047] In operation S220, a first fusion feature of the document page is determined according to the attention features of each of the plurality of modal features.

[0048] In operation S230, a recognition result of the document is determined according to the first fusion feature of the document page.

[0049] The documents in the embodiments of the present disclosure are of various types, and may be documents of various types published online by users, such as plain text type, table type, image type, etc.

[0050] A document usually includes at least one document page, such as a homepage. Each document page may contain multimodal data. Modal features of at least two modalities may be extracted from the multimodal data. For example, modal features of at least one modality may be extracted from text data in the multimodal data, such as title features and semantic features.

[0051] In an illustrative embodiment, the modal features of each modality may be extracted using a variety of methods such as a deep learning neural network, a convolutional neural network, and a visual-text pre-training model.

[0052] The attention mechanism can be used to further extract features from each modality feature, extract information related to the document recognition task under the modality, and obtain the attention features of each modality. For example, at least one of the following types of attention mechanisms can be used for feature extraction, such as self-attention mechanism, multi-head self-attention mechanism, channel attention mechanism, spatial attention mechanism, adaptive attention mechanism, etc.

[0053] The attention features of the multiple modal features can be fused in a variety of ways to obtain the first fused feature. For example, the attention features of the multiple modal features can be fused once; or, the attention features of the multiple modal features can be fused multiple times; or, after fusing the attention features of the multiple modal features, feature enhancement is performed; or, after fusing the attention features of the multiple modal features, the fused features can be fused again with the attention features of the multiple modal features.

[0054] The first fusion feature refers to a feature that integrates multiple modes of the document page. For example, the first fusion feature may be a feature obtained by integrating multiple modes such as semantics, title, and layout.

[0055] The document recognition of the embodiment of the present disclosure may be a binary classification task. After determining the first fusion feature of the document page, a fully connected network, a convolutional neural network or a classifier may be used to classify the first fusion feature to obtain a recognition result of the document.

[0056] For example, the recognition result can be: high-quality document or low-quality document. High-quality / low-quality documents are usually distinguished from the perspectives of whether the layout is reasonable, whether the typesetting is reasonable, whether the word order is wrong, whether there are typos, etc.

[0057] In an illustrative embodiment, the document recognition method can be implemented by a document recognition model. For example, multimodal data of a document page in a document is input into the document recognition model, and the model extracts features of each modality of the multimodal data to obtain features of each modality; extracts attention features from the modal features of each modality; fuses the attention features of each modality to obtain a first fused feature; and classifies the first fused feature to output a recognition result of the document.

[0058] In the embodiment of the present disclosure, multiple modal features are obtained by extracting features of each modality from the multimodal data of the document page in the document; the first fusion feature of the document page is determined based on the attention features of each of the multiple modal features; and the recognition result of the document is determined based on the first fusion feature of the document page. Since the embodiment of the present disclosure extracts attention features for each modal feature, the feature quality of a single modality is further optimized; then, the attention features of each modality are fused into the first fusion feature, so that the recognition result obtained based on the first fusion feature of the fusion of multiple modalities can fully couple the features between multiple modalities, thereby improving the document recognition accuracy.

[0059] Reference below Figure 3~Figure 6 , combined with specific embodiments Figure 2 The method shown is further explained.

[0060] Figure 3 The following schematically shows an application scenario diagram for determining a recognition result of a document according to an embodiment of the present disclosure.

[0061] like Figure 3As shown, in the embodiment 300, for a certain document page 301 in the document, the multimodal data 302 of the document page 301 is subjected to feature extraction of each modality, and modal features of M modalities can be obtained, such as modal feature 1 303_1, modal feature 2 303_2, ..., modal feature M 303_M. For each modal feature, an attention feature can be extracted from it using an attention mechanism, such as extracting attention feature 1 304_1 from modal feature 1 303_1, extracting attention feature 2 304_2 from modal feature 2 303_2, ..., extracting attention feature M 304_M from modal feature M 303_M. By fusing attention feature 1 304_1, attention feature 2 304_2, ..., attention feature M 304_M, a first fused feature 305 is obtained. Further, the first fused feature 305 is identified to obtain a recognition result 306 of the document.

[0062] For example, in a document recognition scenario, the M modal features mentioned above may be layout features, semantic features, and title features.

[0063] In the document recognition scenario, high-quality documents in domain S do not belong to high-quality documents in domain B. Therefore, the embodiments of the present disclosure use the title as a modality when performing document recognition and integrate it into document recognition. In particular, in the document recommendation scenario, when performing document recognition on multiple documents in a document library, it is necessary not only to identify high-quality documents for users, but also to recommend high-quality documents that meet their desired domains to users. At this time, using the title as a modality can better perform document recognition and document domain-based recommendation.

[0064] The embodiments of the present disclosure further optimize the feature quality of a single modality by extracting attention features from layout features, semantic features, and title features; then, the attention features of each modality are fused into a first fused feature, so that the document recognition task can fully couple the information associations between multiple modalities, especially the relationship between the title modality and the semantic and layout modalities, thereby improving the document recognition accuracy.

[0065] According to an embodiment of the present disclosure, for operation S220, a first fusion feature of the document page is determined based on the attention features of each of the multiple modal features, including: determining a second fusion feature based on the attention features of each of the multiple modal features; and determining a first fusion feature based on the second fusion feature and the attention features of each of the multiple modal features.

[0066] In the embodiments of the present disclosure, considering the complementarity of multiple modal features, the attention features of the multiple modalities can be added together to obtain a second fused feature. Alternatively, after adding the attention features of the multiple modalities, further feature extraction is performed on the added features to obtain a second fused feature with strong robustness.

[0067] For example, the added features can be input into a multi-layer neural network such as a multilayer perceptron (MLP) to perform feature extraction on the added features to obtain a second fused feature after feature enhancement.

[0068] For the second fusion feature that preliminarily couples multiple modalities, the fusion method is relatively simple and may not be able to fully couple the features between multiple modalities. Therefore, after obtaining the preliminarily coupled second fusion feature, the attention features of each modality are used again to fuse the second fusion feature with the attention features of each of the multiple modal features to obtain the first fusion feature that fully couples multiple modalities.

[0069] For example, the second fusion feature and the attention features of each of the multiple modal features can be directly added to obtain the first fusion feature.

[0070] Figure 4A A schematic diagram of a scenario for determining a first fusion feature according to an embodiment of the present disclosure is schematically shown.

[0071] like Figure 4A As shown, in embodiment 400A, attention feature 1 402_1 is extracted from modal feature 1 401_1, attention feature 2 402_2 is extracted from modal feature 2 401_2, ..., and attention feature M 402_M is extracted from modal feature M 401_M. By fusing attention feature 1 402_1, attention feature 2 402_2, ..., attention feature M 402_M, a second fused feature 403 is obtained. The second fused feature 403 is then fused with the M attention features to obtain a first fused feature 404.

[0072] For example, taking the modal features as layout features, semantic features, and title features, they are f i 、f t 、f b , the attention features of the three modal features are A i , A t , A b .in, , , , b, c and emb represent batch, number of channels and encoding length respectively. The second fusion feature can be MLP (A i +A t +A b ) represents the output of adding the attention features of the three modal features and inputting them into the MLP. The first fusion feature is: (A i +A t +A b)+MLP(A i +A t +A b ).

[0073] The embodiment of the present disclosure uses at least two fusion operations to fully couple the attention features of multiple modalities. Compared with the existing method of fusing features by splicing / weighting, the embodiment of the present disclosure can improve the accuracy of document recognition. In addition, adding the attention features of a single modality and the second fusion feature that fuses the attention features of multiple modalities can better achieve multi-modal fusion.

[0074] According to an embodiment of the present disclosure, a first fusion feature is determined based on a second fusion feature and respective attention features of a plurality of modal features, including: fusing respective attention features of a plurality of modal features with the second fusion feature to obtain a third fusion feature; and performing feature enhancement on the third fusion feature to obtain a first fusion feature.

[0075] In an exemplary embodiment, the attention features of the multiple modal features and the second fusion feature may be added together to obtain a third fusion feature, and then the third fusion feature may be enhanced by multi-layer convolution to obtain the first fusion feature.

[0076] Figure 4B FIG. 2 schematically shows a scene diagram of determining a first fusion feature according to another embodiment of the present disclosure. Figure 4B As shown, in embodiment 400B, M modal features, M attention features, second fusion features 403 and first fusion features 404 are Figure 4A The same or similar parts are not described here.

[0077] After obtaining the second fused feature 403 based on the attention feature 1 402_1, the attention feature 2 402_2, ..., the attention feature M 402_M, the second fused feature 403 is added to the attention feature 1 402_1, the attention feature 2 402_2, ..., the attention feature M 402_M to obtain the third fused feature 405, and then the third fused feature 405 is enhanced to obtain the first fused feature 404.

[0078] In another exemplary embodiment, the third fused feature can be enhanced by at least one layer of Transformer structure. For example, the third fused feature is used as the input of the Transformer structure, the input of each layer of Transformer structure is the output of the previous layer of Transformer structure, and the output of the last layer of Transformer structure is the first fused feature.

[0079] The embodiments of the present disclosure further enhance the features of the third fusion feature that fuses multiple modalities, so that document recognition can be better achieved based on the fused first fusion feature, thereby improving the accuracy of document recognition.

[0080] According to an embodiment of the present disclosure, the modal features include layout features, semantic features and title features; the document recognition method also includes: replacing the attention feature of the title feature with the first fusion feature, and determining a new first fusion feature based on the first fusion feature, the attention feature of the layout feature, and the attention feature of the semantic feature.

[0081] The first fusion feature can also be determined through a multi-layer adaptive attention mechanism (Adaptive-Attention, A-Atn) structure, such as 3 A-Atn layers. Each A-Atn layer includes M inputs and M outputs. For example, the input of the first A-Atn layer is M modal features, and the output is the first fusion feature and the attention feature of (M-1) modal features. The input of each subsequent A-Atn layer is the output of the previous A-Atn layer and the attention feature of (M-1) modal features.

[0082] Figure 4C The following schematically shows a scene diagram of determining the first fusion feature according to another embodiment of the present disclosure. Figure 4C As shown, in this embodiment 400C, the process of obtaining the second fusion feature 403 based on the attention feature 1 402_1, the attention feature 2 402_2, ..., the attention feature M402_M is the same as Figure 4A similar.

[0083] The first fused feature 404 is used to replace the attention feature 1 402_1 of the modal feature 1, and then the first fused feature 404 is fused with the attention feature 2 402_2, ..., attention feature M 402_M to obtain a new first fused feature 406.

[0084] Considering that the title carries less information in the document recognition scenario, and the layout and semantics contain richer features, for the modal fusion part, while strengthening the information association between multiple modalities, the title features in the subsequent A-Atn layer can be replaced with the first fusion features output by the previous layer, while retaining the attention features of the layout features and the attention features of the semantic features.

[0085] The embodiments of the present disclosure can better mine the potential features of semantic features and layout features by replacing the attention features of the title modality with the first fusion features that fuse multiple modalities.

[0086] According to an embodiment of the present disclosure, multimodal data includes image data and text data; feature extraction of each modality is performed on the multimodal data of a document page in a document to obtain multiple modal features, including: using an image expert model to extract features from image data to obtain layout features; using a long text expert model to extract features from text data to obtain semantic features; using a short text expert model to extract features from text data to obtain title features.

[0087] In this embodiment, an expert model is built to extract the modal features of the three modalities respectively, paving the way for subsequent feature fusion.

[0088] Image data can be a screenshot of a document page in a document, and text data is data that has a binding relationship with the document page. For example, for a plain text document, the text data can be directly obtained from the document, and a binding relationship can be established with the corresponding document page through the position of the text data in the document. Alternatively, the text data can also be obtained by extracting text from the image data, and the text data can be directly bound based on the document page that is the extraction source.

[0089] In addition, for a document, the title may be the main title of the entire document or a subtitle within the document, and thus, the title feature may be obtained from at least one document page in the document.

[0090] For online document recognition, considering the online deployment problem, the embodiments of the present disclosure utilize an expert model with strong feature extraction capability for each modality to perform feature extraction for each modality.

[0091] For example, the layout features of a document can be extracted using an image expert model based on the Vision Transformer (ViT). Layout features can be understood as the typesetting features of a document. Semantic features can be extracted using a long text expert model that has strong feature extraction capabilities for long texts, such as the Long Bert model. Semantic features can be understood as features of dimensions such as typos, missing characters, and garbled characters in a document. For title features, since the title itself is short, a short text expert model can be used to extract features from text data, such as the Bert model. Title features can be understood as the semantics of the title.

[0092] The embodiments of the present disclosure perform feature extraction on text and image data respectively based on the expert model, so that the three models can maintain their original powerful feature extraction capabilities in a single modality and ensure the accuracy of the modal features.

[0093] Figure 5The exemplary system architecture of the document recognition model according to an embodiment of the present disclosure is schematically shown. The document recognition model includes a feature extraction layer, a feature fusion layer and an output layer, and the output layer may include a linear layer to map the first fused feature to two types of probabilities, high quality and non-high quality. Embodiment 500 schematically shows the feature extraction layer and the feature fusion layer in the document recognition model.

[0094] like Figure 5 As shown, the feature extraction layer 501 includes an image expert model M1, a short text expert model M2, and a long text expert model M3. The image expert model M1 extracts features from the image data in the multimodal data and outputs a layout feature f i ; The short text expert model M2 extracts features from the text data in the multimodal data and outputs the title feature f b ; The long text expert model M3 extracts features from the text data in the multimodal data and outputs semantic features f t .

[0095] The feature fusion layer includes multiple layers of adaptive attention mechanism structures, such as the first adaptive attention mechanism structure, namely the first A-Atn layer 502_1 and other A-Atn layers 502_2.

[0096] In the first A-Atn layer 502_1, the layout features f i 、Title Features b , semantic features f t For example, the layout feature f is added through the convolutional network i 、Title Features b , semantic features f t Mapped to the corresponding Q, K, and V matrices respectively.

[0097] For example, for the layout feature f i , the mapped matrix can be expressed as Q i , K i 、V i , for the title feature f b , the mapped matrix can be expressed as Q b , K b 、V b , for the semantic feature f t , the mapped matrix can be expressed as Q t , K t 、V t .

[0098] For the layout feature f i , generate layout features f i Attention feature A i As shown in formula (1):

[0099] (1)

[0100] in, is the scaling factor, d is the number of dimensions, and the semantic feature f t Attention feature A t 、Title Features b Attention feature A b similar.

[0101] Add the attention features of the three modal features to get A=A i +A t +A b , and feed it into the MLP to obtain the output f of the first A-Atn layer 502_1 m =(A i +A t +A b )+MLP(A i +A t +A b ), and the layout features f i Attention feature A i and semantic features f t Attention feature A t After that, the attention feature A i , Attention Feature A t , and the first fusion feature f m As the input of the other A-Atn layer 502_2, to generate a new first fusion feature. Afterwards, the output layer takes the f output of the last A-Atn layer as m Classification is performed without the need for attention feature A on the output i , Attention Feature A t The final output is .

[0102] According to an embodiment of the present disclosure, a recognition result of a document is determined based on a first fusion feature of a document page, including: determining a page recognition result of each of the multiple document pages in the document based on the first fusion feature of each of the multiple document pages, wherein the multiple document pages are determined based on the number of pages of the document; and determining a recognition result of the document based on the page recognition results of each of the multiple document pages.

[0103] In an embodiment of the present disclosure, a document generally includes at least one document page, and therefore, page recognition results of multiple document pages in the document may be integrated to determine a recognition result of the entire document.

[0104] In an exemplary embodiment, the multimodal data of each document page in the document may be processed to obtain first fused data of each document page, and the first fused data may be classified to obtain page recognition results of each of the plurality of document pages. Then, the recognition result of the document may be determined based on the page recognition results of each of the plurality of document pages.

[0105] In another exemplary embodiment, considering that different document pages contribute differently to document recognition, the multimodal data of each document page in the document can be processed to obtain first fused data of each document page, and the page recognition results of each of the multiple document pages can be obtained based on the first fused data. Then, the contribution of each document page to document recognition is used as the weight of the corresponding document page, and the page recognition results of each of the multiple document pages are weighted and summed to obtain the recognition result of the document.

[0106] For a document library containing hundreds of millions of documents, if high-quality document identification is achieved through full-text content analysis, the resource consumption will be extremely large.

[0107] In another embodiment, multiple document pages in the document may be randomly selected, and the multimodal data of each document page may be processed to obtain the first fused data of each document page, and the page recognition result of each document page may be obtained based on the first fused data. The recognition result of the document may be determined by combining the page recognition results of the multiple document pages. Of course, it is understandable that the weighted sum may also be calculated according to the contribution of different document pages.

[0108] The embodiments of the present disclosure can recognize a document more comprehensively and completely by integrating the page recognition results of multiple document pages in the document.

[0109] Figure 6 A schematic diagram schematically shows a scenario of determining a recognition result based on multiple document pages according to an embodiment of the present disclosure.

[0110] like Figure 6 As shown, in embodiment 600, for document 601, three document pages are selected for page recognition. For example, the multimodal data of document page 1 602_1 is processed to obtain page recognition result 1 603_1, the multimodal data of document page 2 602_2 is processed to obtain page recognition result 2 603_2, and the multimodal data of document page 3 602_3 is processed to obtain page recognition result 3 603_3. By calculating the average of page recognition result 1 603_1, page recognition result 2 603_2, and page recognition result 3 603_3, the recognition result 604 of document 601 is obtained.

[0111] However, the document library includes a large number of documents, but the page distribution of multiple documents varies greatly, ranging from a few pages to hundreds of pages. If the document pages are not distinguished, the document recognition accuracy may be affected.

[0112] According to an embodiment of the present disclosure, the document recognition method also includes: when the number of pages of a document is greater than a predetermined threshold, determining the document first page, document second page and document last page of the document as multiple document pages; when the number of pages of a document is less than or equal to a predetermined threshold, determining each document page of the document as multiple document pages.

[0113] This paper analyzes the data of the document library and finds that the largest proportion of problems are on the first page, the second page, and the last page.

[0114] Therefore, for a document with a page number greater than a predetermined threshold, that is, a long document, the present disclosure determines the document homepage, the document second page, and the document last page as multiple document pages, and combines the page recognition results of the document homepage, the document second page, and the document last page to determine the recognition result of the entire document. For a document with a page number less than a predetermined threshold, such as 3 pages, each document page of the document can be determined as multiple document pages.

[0115] The embodiment of the present disclosure selects multiple document pages for document recognition based on the number of pages, and integrates the page recognition results of the multiple document pages to obtain the recognition result of the document, which can reduce the amount of data for recognition calculation and improve recognition accuracy.

[0116] Figure 7 A scenario diagram schematically illustrates a method for training a document recognition model according to an embodiment of the present disclosure.

[0117] like Figure 7 As shown, the method 700 includes operations S710 to S720.

[0118] In operation S710, a training sample set is acquired.

[0119] In operation S720, the document recognition model is trained using sample documents in the training sample set, so that the document recognition model performs the following operations, such as operations S721 to S723:

[0120] In operation S721, feature extraction of each modality is performed on multimodal data of a document page in a sample document to obtain a plurality of modality features.

[0121] In operation S722, a first fusion feature of the document page is determined according to the attention features of each of the plurality of modal features.

[0122] In operation S723, a recognition result of the sample document is determined according to the first fusion feature of the document page.

[0123] In this embodiment, the training sample set includes multiple sample documents, and the sample documents are similar to the definition of the above documents, which are only used to distinguish between different embodiments. Operation S721 to Operation S723 are as specific as Operation S210 to Operation S230 above, and will not be repeated here.

[0124] According to an embodiment of the present disclosure, the document recognition model can use a cross entropy loss function to calculate the loss, and when the loss value converges, a trained document recognition model is obtained. The cross entropy loss function is as follows (2):

[0125] (2);

[0126] Among them, L is the loss function, y pred is the label, y i is the page recognition result of the i-th sample document page output by the document recognition model, and N is the number of samples of the sample document page.

[0127] The embodiments of the present disclosure implement document recognition by training a document recognition model. The document recognition model can, in the process of using multimodal data for document recognition, fully couple multimodal features by fusing the respective attention features of multiple modalities to improve document recognition performance.

[0128] According to an embodiment of the present disclosure, the sample documents include positive sample documents and sampling sample documents. Acquiring a training sample set includes: determining at least one target document publisher based on the flow information of the document publisher; acquiring the positive sample document from the historical documents published by at least one of the target document publishers; and acquiring the sampling sample document from the document library.

[0129] Document publishers refer to users / publishers who have published documents in the past and whose documents are included in the document library. Traffic information can include the number of likes, views, subscriptions, and shares.

[0130] The target document publisher may be a publisher whose traffic information meets predetermined conditions, such as at least one of the number of likes, views, subscriptions, and shares is greater than a traffic threshold. Thus, the historical documents published by the target document publisher may be regarded as positive sample documents that tend to be high-quality documents.

[0131] In an exemplary embodiment, multiple positive sample documents may be obtained from historical documents published by at least one target document publisher based on the number of training data sets, that is, directional sampling. Sampled sample documents may be randomly selected from a document library, that is, random sampling.

[0132] Considering that when obtaining the training sample set in the training stage, the existing method is usually to randomly select the training sample set from the document library. However, the quality distribution of high-quality, low-quality and ordinary documents in the document library is relatively uneven, among which high-quality only accounts for about 16% of the total sample volume. Therefore, random sampling will produce serious sample imbalance, resulting in overfitting of the document recognition model and selecting a large number of low-quality or ordinary samples. Therefore, the embodiments of the present disclosure can increase the probability of selecting positive sample documents through directional sampling, and at the same time, randomly obtain sampled sample documents from the document library to ensure the generalization of the model.

[0133] According to an embodiment of the present disclosure, training a document recognition model using sample documents in a training sample set includes: while freezing the feature extraction parameters of the document recognition model, adjusting the fusion parameters of the document recognition model based on the recognition results of the sample documents to train the document recognition model; wherein the document recognition model extracts features of each modality for multimodal data of document pages in the sample documents based on the feature extraction parameters; and the document recognition model determines a first sample fusion feature based on the fusion parameters and according to the attention features of each of the multiple modal features.

[0134] The document recognition model includes a feature extraction layer, which is used to extract features of each modality from multimodal data of document pages in the sample document to obtain multiple sample modality features.

[0135] When training a document recognition model, the feature extraction parameters of the feature extraction layer are frozen so that the feature extraction layer can maintain the original single-modal feature extraction performance when training the document recognition model. As a result, the fusion parameter tuning can be completed more quickly to obtain a trained document recognition model.

[0136] For example, the sample document may include at least one sample document page, and the multimodal data of the sample document page includes sample image data and sample text data. The feature extraction layer includes an image expert model, a long text expert model, and a short text expert model. When training the document recognition model, the parameters in the image expert model, the long text expert model, and the short text expert model are frozen, so that the document recognition model can use the image expert model to extract features from the sample image data to obtain sample layout features; use the long text expert model to extract features from the sample text data to obtain sample semantic features; and use the short text expert model to extract features from the sample text data to obtain sample title features.

[0137] The document recognition model includes a feature fusion layer for determining a first sample fusion feature of a document page according to the attention features of each of the multiple modal features. During training, the document recognition model can determine a first sample fusion feature of a sample document page according to the attention features of each of the multiple sample modal features.

[0138] In an illustrative embodiment, the document recognition model can determine a second sample fusion feature based on the fusion parameter and according to the attention features of each of the multiple sample modal features; and determine the first sample fusion feature based on the second sample fusion feature and the attention features of each of the multiple sample modal features.

[0139] In an illustrative embodiment, for determining the first sample fusion feature based on the second sample fusion feature and the respective attention features of multiple sample modal features: the document recognition model can also fuse the respective attention features of multiple sample modal features and the second sample fusion feature based on the fusion parameter to obtain a third sample fusion feature; and perform feature enhancement on the third sample fusion feature to obtain the first sample fusion feature.

[0140] In an illustrative embodiment, the attention feature of the sample title feature can also be replaced by the first sample fusion feature based on the fusion parameter, and a new first sample fusion feature can be determined based on the first sample fusion feature, the attention feature of the sample layout feature, and the attention feature of the sample semantic feature.

[0141] In an illustrative embodiment, during the training phase, for a plurality of sample document pages of a sample document, the document recognition model may determine a sample page recognition result of each sample document page according to the first sample fusion feature of each sample document page, wherein the plurality of sample document pages are determined according to the number of pages of the sample document. The sample recognition result of the sample document is determined according to the sample page recognition results of the plurality of sample document pages.

[0142] According to an embodiment of the present disclosure, it also includes: when the number of pages of the sample document is greater than a predetermined threshold, determining a sample document first page, a sample document second page, and a sample document last page of the sample document as multiple sample document pages; when the number of pages of the sample document is less than or equal to the predetermined threshold, determining each sample document page of the sample document as multiple sample document pages.

[0143] Figure 8 A block diagram of a document recognition device according to an embodiment of the present disclosure is schematically shown.

[0144] like Figure 8 As shown, the document recognition device 800 includes a feature extraction module 810 , a fusion module 820 , and a recognition module 830 .

[0145] The feature extraction module 810 is used to extract features of each modality from multimodal data of a document page in a document to obtain multiple modality features.

[0146] The fusion module 820 is used to determine a first fusion feature of the document page according to the attention features of each of the multiple modal features.

[0147] The recognition module 830 is used to determine the recognition result of the document according to the first fusion feature of the document page.

[0148] According to an embodiment of the present disclosure, the fusion module includes: a first fusion submodule and a second fusion submodule.

[0149] The first fusion submodule is used to determine a second fusion feature that fuses multiple modalities according to the attention features of each of the multiple modal features.

[0150] The second fusion submodule is used to determine the first fusion feature according to the second fusion feature and the attention features of each of the multiple modal features.

[0151] According to an embodiment of the present disclosure, the second fusion submodule includes: a first fusion unit and a second fusion unit.

[0152] The first fusion unit is used to fuse the attention features of the plurality of modal features and the second fusion features to obtain a third fusion feature.

[0153] The second fusion unit is used to enhance the third fusion feature to obtain the first fusion feature.

[0154] According to an embodiment of the present disclosure, the modal features include layout features, semantic features and title features. The fusion module also includes: a third fusion submodule, which is used to replace the attention feature of the title feature with the first fusion feature, and determine a new first fusion feature based on the first fusion feature, the attention feature of the layout feature, and the attention feature of the semantic feature.

[0155] According to an embodiment of the present disclosure, the multimodal data includes image data and text data. The feature extraction module includes: a first feature extraction submodule, a second feature extraction submodule, and a third feature extraction submodule.

[0156] The first feature extraction submodule is used to extract features from image data using an image expert model to obtain layout features.

[0157] The second feature extraction submodule is used to extract features from text data using a long text expert model to obtain semantic features.

[0158] The third feature extraction submodule is used to extract features from text data using the short text expert model to obtain title features.

[0159] According to an embodiment of the present disclosure, the identification module includes: a first identification submodule and a second identification submodule.

[0160] The first recognition submodule is used to determine page recognition results of the multiple document pages according to the first fusion features of the multiple document pages in the document, wherein the multiple document pages are determined according to the number of pages of the document.

[0161] The second recognition submodule is used to determine the recognition result of the document according to the page recognition results of each of the multiple document pages.

[0162] According to an embodiment of the present disclosure, the document recognition device further includes: a first quantity determination module and a second quantity determination module.

[0163] The first quantity determination module is used to determine a document first page, a document second page, and a document last page of the document as a plurality of document pages when the number of pages of the document is greater than a predetermined threshold.

[0164] The second quantity determination module is used to determine each document page of the document as a plurality of document pages when the number of pages of the document is less than or equal to a predetermined threshold.

[0165] Fig. 9 A block diagram of a training device for a document recognition model according to an embodiment of the present disclosure is schematically shown.

[0166] like Fig. 9 As shown, the document recognition model training device 900 includes: an acquisition module 910 and a training module 920.

[0167] The acquisition module 910 is used to acquire a training sample set, wherein the training sample set includes a plurality of sample documents.

[0168] The training module 920 is used to train the document recognition model using the sample documents in the training sample set, so that the document recognition model performs the following operations: extracting features of each modality of the multimodal data of the document page in the sample document to obtain multiple modal features; determining the first fusion feature of the document page based on the attention features of each of the multiple modal features; and determining the recognition result of the sample document based on the first fusion feature of the document page.

[0169] According to an embodiment of the present disclosure, the sample document includes a positive sample document and a sampling sample document. The acquisition module includes: a publisher determination submodule, a first acquisition submodule and a second acquisition submodule.

[0170] The publisher determination submodule is used to determine at least one target document publisher based on the flow information of the document publisher.

[0171] The first acquisition submodule is used to acquire positive sample documents from historical documents published by at least one target document publisher.

[0172] The second acquisition submodule is used to acquire sample documents from the document library.

[0173] According to an embodiment of the present disclosure, the training module includes: a parameter adjustment module, which is used to adjust the fusion parameters of the document recognition model based on the recognition results of the sample document to train the document recognition model while freezing the feature extraction parameters of the document recognition model; wherein the document recognition model extracts features of each modality of the multimodal data of the document page in the sample document based on the feature extraction parameters; the document recognition model determines the first fusion feature based on the fusion parameters and according to the attention features of each of the multiple modal features.

[0174] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0175] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0176] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0177] According to an embodiment of the present disclosure, a computer program product includes a computer program, and the computer program implements the above method when executed by a processor.

[0178] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0179] Fig.10 A block diagram of an electronic device suitable for implementing a document recognition method and a training method for a document recognition model according to an embodiment of the present disclosure is schematically shown.

[0180] like Fig.10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0181] A number of components in the device 1000 are connected to an input / output (I / O) interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0182] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a document recognition method or a training method for a document recognition model. For example, in some embodiments, the document recognition method or the training method for a document recognition model may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the document recognition method or the training method for the document recognition model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured as a document recognition method or a document recognition model training method in any other appropriate manner (eg, by means of firmware).

[0183] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0185] In the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0187] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0188] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0189] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0190] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A document recognition method, comprising: Extracting features of each modality from multimodal data of document pages in the document to obtain multiple modality features; Determining a first fusion feature of the document page according to respective attention features of the plurality of modal features; as well as A recognition result of the document is determined according to the first fusion feature of the document page.

2. The method according to claim 1, wherein: The determining the first fusion feature of the document page according to the attention features of each of the plurality of modal features includes: Determining a second fusion feature according to the attention features of each of the plurality of modal features; and The first fusion feature is determined according to the second fusion feature and the attention features of each of the plurality of modal features.

3. The method according to claim 2, wherein: The determining the first fusion feature according to the second fusion feature and the attention features of each of the plurality of modal features comprises: fusing the attention features of each of the plurality of modal features and the second fused feature to obtain a third fused feature; and Feature enhancement is performed on the third fused feature to obtain the first fused feature.

4. The method according to any one of claims 1 to 3, wherein The modal features include layout features, semantic features and title features; The method further comprises: The attention feature of the title feature is replaced by the first fused feature, and a new first fused feature is determined according to the first fused feature, the attention feature of the layout feature, and the attention feature of the semantic feature.

5. The method according to any one of claims 1 to 4, wherein: The multimodal data includes image data and text data; the multimodal data of the document page in the document is subjected to feature extraction of each modality to obtain multiple modal features, including: Using an image expert model to extract features from the image data to obtain the layout features; Using a long text expert model to extract features from the text data to obtain the semantic features; The short text expert model is used to extract features from the text data to obtain the title features.

6. The method according to claim 1, wherein: The determining the recognition result of the document according to the first fusion feature of the document page includes: Determining page recognition results of each of the plurality of document pages in the document according to the first fusion features of each of the plurality of document pages in the document, wherein the plurality of document pages are determined according to the number of pages of the document; The recognition result of the document is determined according to the page recognition results of each of the plurality of document pages.

7. The method according to claim 6, further comprising: When the number of pages of the document is greater than a predetermined threshold, determining a document first page, a document second page, and a document last page of the document as a plurality of document pages; In a case where the number of pages of the document is less than or equal to the predetermined threshold, each of the document pages of the document is determined as a plurality of the document pages.

8. A method for training a document recognition model, comprising: Acquire a training sample set, wherein the training sample set includes a plurality of sample documents; The document recognition model is trained using the sample documents in the training sample set, so that the document recognition model performs the following operations: Extracting features of each modality from multimodal data of document pages in the sample document to obtain multiple modality features; Determining a first fusion feature of the document page according to the attention features of each of the plurality of modal features; and A recognition result of the sample document is determined according to the first fusion feature of the document page.

9. The method according to claim 8, wherein: The sample documents include positive sample documents and sampling sample documents; and obtaining a training sample set includes: Determining at least one target document publisher based on the traffic information of the document publisher; Acquire the positive sample document from historical documents published by at least one of the target document publishers; The sampling sample document is obtained from the document library.

10. The method according to claim 8, wherein: The using of sample documents in the training sample set to train the document recognition model comprises: In a case where the feature extraction parameters of the document recognition model are frozen, adjusting the fusion parameters of the document recognition model based on the recognition result of the sample document to train the document recognition model; Among them, the document recognition model extracts features of each modality of the multimodal data of the document page in the sample document based on the feature extraction parameters; the document recognition model determines the first fusion feature based on the fusion parameters and according to the attention features of each of the multiple modal features.

11. A document recognition device, comprising: A feature extraction module is used to extract features of each modality from multimodal data of a document page in a document to obtain multiple modal features; A fusion module, configured to determine a first fusion feature of the document page according to respective attention features of the plurality of modal features; as well as The recognition module is used to determine the recognition result of the document according to the first fusion feature of the document page.

12. The device according to claim 10, wherein: The fusion module includes: A first fusion submodule is used to determine a second fusion feature that fuses the multiple modalities according to the attention features of each of the multiple modal features; and The second fusion submodule is used to determine the first fusion feature according to the second fusion feature and the attention features of each of the plurality of modal features.

13. The device according to claim 12, wherein: The second fusion submodule includes: A first fusion unit is used to fuse the attention features of each of the plurality of modal features and the second fusion feature to obtain a third fusion feature; and The second fusion unit is used to perform feature enhancement on the third fusion feature to obtain the first fusion feature.

14. The device according to any one of claims 11 to 13, wherein: The modal features include layout features, semantic features and title features; The fusion module also includes: The third fusion submodule is used to replace the attention feature of the title feature with the first fusion feature, and determine a new first fusion feature according to the first fusion feature, the attention feature of the layout feature, and the attention feature of the semantic feature.

15. The device according to any one of claims 11 to 14, wherein: The multimodal data includes image data and text data; the feature extraction module includes: A first feature extraction submodule is used to extract features from the image data using an image expert model to obtain the layout features; A second feature extraction submodule is used to extract features from the text data using a long text expert model to obtain the semantic features; The third feature extraction submodule is used to extract features from the text data using a short text expert model to obtain the title features.

16. The device according to claim 11, wherein The identification module comprises: A first recognition submodule, configured to determine a page recognition result of each of the plurality of document pages in the document according to the first fusion feature of each of the plurality of document pages in the document, wherein the plurality of document pages is determined according to the number of pages of the document; The second recognition submodule is used to determine the recognition result of the document according to the page recognition results of each of the plurality of document pages.

17. The device according to claim 16, wherein the document recognition device further comprises: A first quantity determination module, configured to determine a document first page, a document second page, and a document last page of the document as a plurality of document pages when the number of pages of the document is greater than a predetermined threshold; The second quantity determination module is configured to determine each document page of the document as a plurality of document pages when the number of pages of the document is less than or equal to the predetermined threshold.

18. A training device for a document recognition model, comprising: An acquisition module, used to acquire a training sample set, wherein the training sample set includes a plurality of sample documents; The training module is used to train the document recognition model using the sample documents in the training sample set, so that the document recognition model performs the following operations: Extracting features of each modality from multimodal data of document pages in the sample document to obtain multiple modality features; Determining a first fusion feature of the document page according to the attention features of each of the plurality of modal features; and A recognition result of the sample document is determined according to the first fusion feature of the document page.

19. The device according to claim 18, wherein: The sample documents include positive sample documents and sampling sample documents; the acquisition module includes: A publisher determination submodule, used to determine at least one target document publisher based on the flow information of the document publisher; A first acquisition submodule, configured to acquire the positive sample document from at least one historical document published by the target document publisher; The second acquisition submodule is used to acquire the sample document from the document library.

20. The device according to claim 18, wherein The training module includes: A parameter adjustment module, configured to adjust the fusion parameters of the document recognition model based on the recognition result of the sample document in order to train the document recognition model while freezing the feature extraction parameters of the document recognition model; Among them, the document recognition model extracts features of each modality of the multimodal data of the document page in the sample document based on the feature extraction parameters; the document recognition model determines the first fusion feature based on the fusion parameters and according to the attention features of each of the multiple modal features.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images