Method, device and related equipment for restoring paragraph structure of scanned documents based on multimodality

Abstract

The embodiment of the present invention discloses a method, an apparatus and related equipment for restoring the paragraph structure of a scanned document based on multimodality. The method recognizes text scan data through a pre-trained OCR model; removes text boxes without text content, and arranges them in sequence according to the coordinate positions of the text boxes, and annotates each text box; takes out the image area where the corresponding text box is located according to each coordinate position; encodes and fuses the text content and image data through a sequence classification model, and predicts the classification result of each text box; reorganizes the paragraphs according to the classification result; combines the text boxes recognized by OCR into a text box sequence, annotates the text box sequence, and integrates the image data taken out of the corresponding text box, adopts a sequence annotation task method, and predicts the annotation probability corresponding to each text box in the sequence through attention fusion and recurrent neural network, and then decodes and reorganizes, so as to obtain a paragraph structure with correct meaning.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a method, device and related equipment for restoring the paragraph structure of a scanned document based on multimodality. Background Art

[0002] OCR (Optical Character Recognition) refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then translates the shapes into computer text using character recognition methods; that is, for printed characters, an optical method is used to convert the text in a paper document into a black and white dot matrix image file, and recognition software is used to convert the text in the image into text format for further editing and processing by word processing software.

[0003] Nowadays, many text information extraction tasks are based on OCR recognition results, such as extracting information such as plaintiffs, defendants, and litigation requests from scanned legal documents. To extract information from scanned documents, the OCR engine is often used to locate and recognize text first, and then the recognized text is analyzed and processed by natural language. Therefore, the accuracy of extracting relevant information from legal documents depends to a certain extent on the OCR recognition results.

[0004] OCR recognition is to detect text in the image from top to bottom and then segment the text area for recognition. There are the following problems in OCR recognition of scanned documents: the sentences recognized by OCR are independent sentences, and the connection between sentences and the structural relationship between paragraphs are lost. Incomplete paragraph information will have a great impact on subsequent text analysis and processing. For example, a legal document begins with "Plaintiff: xxx Co., Ltd., domiciled at xx floor, room xx, No. xx, xxx Road". In the scanned document, "xx floor, room xx, No. xx" is on the next line of "Plaintiff: xxx Co., Ltd., domiciled at xxx Road". If the OCR recognition result considers these two sentences to be two different paragraphs, part of the address information will be lost, and only "xxx Road" will be left when the address is extracted, resulting in incomplete information and incorrect meaning. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and related equipment for restoring the paragraph structure of a scanned document based on multimodality, aiming to solve the problem of incomplete information and incorrect meaning caused by OCR scanning text extraction technology in the prior art.

[0006] In a first aspect, an embodiment of the present invention provides a method for restoring a paragraph structure of a scanned document based on multimodality, comprising:

[0007] Recognize the text scan data through a pre-trained OCR model to obtain a plurality of text boxes, the text content of each of the text boxes, and the coordinate position of each of the text boxes corresponding to the text scan data;

[0008] Eliminate the text boxes without text content, and arrange the text boxes in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and mark each text box;

[0009] According to each of the coordinate positions, the image area where the corresponding text box is located is taken out to obtain the image data corresponding to each of the text boxes;

[0010] Performing encoding fusion processing on the text content and the image data through a sequence classification model, and predicting the classification result of each text box;

[0011] The paragraphs are reorganized according to the classification results.

[0012] In a second aspect, an embodiment of the present invention provides a multimodal scanned document paragraph structure restoration device, comprising:

[0013] An OCR recognition unit, used to recognize the text scan data through a pre-trained OCR model, and obtain a plurality of text boxes, the text content of each of the text boxes, and the coordinate position of each of the text boxes corresponding to the text scan data;

[0014] The optimization and annotation unit is used to remove the text boxes without text content, and arrange the text boxes in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and annotate each of the text boxes;

[0015] An image acquisition unit, used for taking out the picture area where the corresponding text box is located according to each coordinate position, and obtaining image data corresponding to each text box;

[0016] A prediction classification unit, used for performing encoding fusion processing on the text content and the image data through a sequence classification model, and predicting a classification result of each text box;

[0017] A reorganization unit is used to reorganize the paragraphs according to the classification result.

[0018] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for restoring the paragraph structure of a scanned document based on multimodality as described in the first aspect above is implemented.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the multi-modal scanned document paragraph structure restoration method described in the first aspect above.

[0020] The embodiment of the present invention discloses a method for restoring the paragraph structure of a scanned document based on multimodality. The method comprises the following steps: recognizing text scan data through a pre-trained OCR model to obtain a plurality of text boxes, the text content of each text box and the coordinate position of each text box corresponding to the text scan data; removing the text boxes without text content and arranging them in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and marking each text box; taking out the image area where the corresponding text box is located according to each coordinate position to obtain the image data corresponding to each text box; encoding and fusion processing is performed on the text content and the image data through a sequence classification model, and predicting the classification result of each text box; and reorganizing the paragraphs according to the classification result.

[0021] By combining the text boxes recognized by OCR into a text box sequence, annotating the text box sequence, and integrating the image data corresponding to the text boxes, a sequence labeling task is adopted, and the labeling probability corresponding to each text box in the sequence is obtained through attention fusion and recurrent neural network prediction. After decoding and reorganization, a paragraph structure with the correct meaning can be obtained, which can greatly improve the accuracy of OCR recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0023] Figure 1 A schematic block diagram of a method for restoring a paragraph structure of a scanned document based on multimodality provided in an embodiment of the present invention;

[0024] Figure 2 A schematic flow chart of a method for restoring a paragraph structure of a scanned document based on multimodality provided by an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of a sub-flow of step S104 in the method for restoring the paragraph structure of a scanned document based on multimodality provided in an embodiment of the present invention;

[0026] Figure 4A schematic diagram of a sub-flow of step S201 in the method for restoring the paragraph structure of a scanned document based on multimodality provided in an embodiment of the present invention;

[0027] Figure 5 A schematic block diagram of a multimodal scanned document paragraph structure restoration device provided by an embodiment of the present invention;

[0028] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0031] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0033] See also Figure 1-2 , a method for restoring the paragraph structure of a scanned document based on multimodality, comprising the following steps S101-S105.

[0034] S101: recognizing text scan data through a pre-trained OCR model to obtain a plurality of text boxes, text content of each of the text boxes, and a coordinate position of each of the text boxes corresponding to the text scan data;

[0035] S102: removing the text boxes without text content, arranging the text boxes in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and marking each of the text boxes;

[0036] S103: extracting the picture area where the corresponding text box is located according to each coordinate position, and obtaining image data corresponding to each text box;

[0037] S104: performing encoding fusion processing on the text content and the image data through a sequence classification model, and predicting a classification result of each text box;

[0038] S105: Reorganize the paragraphs according to the classification results.

[0039] In this embodiment, the text boxes recognized by OCR are combined into a text box sequence, and the text box sequence is annotated. At the same time, the image data corresponding to the text boxes are integrated, and a sequence labeling task is adopted. The labeling probability corresponding to each text box in the sequence is obtained through attention fusion and recurrent neural network prediction, and then decoding and reorganization are performed to obtain a paragraph structure with the correct meaning.

[0040] By serializing each text box, there is a contextual relationship between each text box. The current text box will depend on the content of the previous text box, and the current text box will also affect the classification result of the next text box, so that the classified text boxes have the correct meaning and can form a correct paragraph structure.

[0041] In step S101, the text scan data is recognized by a pre-trained OCR model to obtain a plurality of text boxes, the text content of each of the text boxes, and the coordinate position of each of the text boxes corresponding to the text scan data.

[0042] In this step, the text scan data is recognized and processed by a pre-trained OCR optical character recognition model to obtain several text boxes. When the OCR model processes different text scan data, the size and position of the intercepted text box will be different, which is based on the text scan data during training. Therefore, there may be problems such as inappropriate text box interception position. If it is only processed by a simple classification model and each text box is recombined to restore the paragraph result, since the text boxes processed by the classification model method are independent of each other, the context dependency between the contents of the text boxes is ignored during classification, which will lead to deviations in the paragraph structure and text meaning. Therefore, the present application subsequently serializes the text box and other data to solve this technical problem.

[0043] In step S102, the text boxes without text content are removed, and the text boxes are arranged in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and each of the text boxes is marked.

[0044] In this step, text boxes without text content are eliminated and redundant parts are removed to facilitate subsequent processing; by sorting the text boxes based on the coordinate position, the positional relationship of the text boxes can be accurately located, and the dependency relationship between the previous and next text boxes in the sequence can also be used in subsequent processing.

[0045] The coordinates of the upper left corner of each text box are the coordinate positions of each text box.

[0046] After obtaining the text box sequence, each text box is annotated using the BIO annotation mode, where B represents the beginning of a paragraph, I represents the middle of a paragraph, and O represents an independent paragraph.

[0047] In step S103, the picture area where the corresponding text box is located is retrieved according to each coordinate position, and image data corresponding to each text box is obtained.

[0048] In this step, the image area is defined by the coordinates of the upper left corner of the text box and the length and width of the text box.

[0049] In step S104, the text content and the image data are encoded and fused using a sequence classification model, and a classification result of each text box is predicted.

[0050] In this step, a sequence labeling task is adopted to predict the labeling probability corresponding to each text box in the sequence through attention fusion and recurrent neural network, and then decode it.

[0051] In step S105, paragraphs are reorganized according to the classification results.

[0052] In this step, the text boxes are reorganized according to the text box type obtained by decoding in step S104 to obtain a paragraph structure with correct meaning.

[0053] Specifically, a judicial document is used to explain how the OCR model intercepts a text box:

[0054] The applicant for retrial (plaintiff in the first instance, appellant in the second instance): Cai **, male, born on * month * day, 19**, Han nationality, living in ** District, Huaihua City, Hunan Province.

[0055] Applicant (defendant in the first instance, appellant in the second instance): Huaihua *** Co., Ltd., domiciled at No. ***, Renmin South Road, ** District, Huaihua City, Hunan Province.

[0056] Legal representative: Chen**, general manager of the company.

[0057] Litigation representative: Nie **, administrator and person in charge of Huaihua ** Co., Ltd.

[0058] The retrial applicant Cai ** was dissatisfied with the civil judgment No. (2020) Hunan Civil Final ** of the Hunan Provincial Higher People's Court in the case of property rights confirmation dispute with Huaihua ** Co., Ltd. (hereinafter referred to as ** Company), and applied to this court for retrial. This court formed a collegial panel in accordance with the law to conduct a review, and the review has now ended.

[0059] After being intercepted by the OCR model, at least the following two situations may be obtained:

[0060] The first case is a text box that ideally gets a paragraph structure with the correct meaning:

[0061] First text box: Applicant for retrial (plaintiff in the first instance, appellant in the second instance): Cai **, male, born on * month * day, 19**, Han nationality, living in ** District, Huaihua City, Hunan Province.

[0062] Second text box: Applicant (defendant in the first instance, appellant in the second instance): Huaihua *** Co., Ltd., domiciled at No. ***, Renmin South Road, ** District, Huaihua City, Hunan Province.

[0063] Third text box: Legal representative: Chen**, general manager of the company.

[0064] Fourth text box: Litigation representative: Nie **, administrator and person in charge of Huaihua ** Co., Ltd.

[0065] Fifth text box: The retrial applicant Cai ** applied to this court for a retrial because he was dissatisfied with the civil judgment No. (2020) Hunan Minzhong ** of the Higher People's Court of Hunan Province in the case of property rights confirmation dispute with Huaihua ** Co., Ltd. (hereinafter referred to as ** Company). This court formed a collegial panel in accordance with the law to conduct a review, and the review has now ended.

[0066] The second case is a text box that may appear without a paragraph structure that has the correct meaning:

[0067] First text box: Applicant for retrial (plaintiff in the first instance, appellant in the second instance): Cai **, male, born on * month * day, 19**, Han nationality, living in ** District, Huaihua City, Hunan Province.

[0068] Second text box: Applicant (defendant in the first instance, appellant in the second instance): Huaihua *** Co., Ltd., domiciled at No. ***, Renmin South Road, ** District, Huaihua City, Hunan Province.

[0069] Legal representative: Chen**, general manager of the company.

[0070] The third text box: Litigation representative: Nie **, administrator and person in charge of Huaihua ** Co., Ltd.

[0071] The retrial applicant Cai ** had a property rights confirmation dispute with Huaihua ** Co., Ltd. (hereinafter referred to as ** Company);

[0072] Fourth text box: Case, dissatisfied with the civil judgment of the Hunan Provincial Higher People's Court (2020) Xiang Min Zhong No. **, applied to this court for retrial. This court formed a collegial panel to review the case in accordance with the law, and the review has now ended.

[0073] In the above text boxes, the third text box and the fourth text box are both text boxes that do not have a paragraph structure with correct meaning.

[0074] See also Figure 3 In one embodiment, step S104 includes:

[0075] S201: Encode and fuse the text content of each text box with the image data to obtain sentence encoding data corresponding to the fused text box;

[0076] S202: Combining each of the sentence coding data in order to obtain a sentence coding sequence;

[0077] S203: encoding the sentence encoding sequence through a recurrent neural network, and normalizing and mapping the encoded sentence encoding sequence;

[0078] S204: Probability prediction is performed on the normalized sentence encoding sequence through a conditional random field, followed by decoding through a beam search, and each of the text boxes is classified to obtain a classification result.

[0079] In this embodiment, by performing encoding fusion processing on the text boxes one by one, sentence encoding data fused with image data can be obtained, and the sentence encoding data is fused with the dependency relationship between the text boxes.

[0080] The sentence encoding data are then arranged and combined in sequence to further strengthen the dependency relationship.

[0081] The dependency relationship is processed through a recurrent neural network, that is, a bidirectional LSTM model or a GRU model. By integrating the current text box with information from other text boxes and training based on context information, the obtained text box information is made richer and more accurate.

[0082] Conditional random field is referred to as CRF in the technical field, and beam search is referred to as beamsearch in the technical field.

[0083] Among them, the classification results adopt the BIO labeling mode.

[0084] Specifically, the second case after the above-mentioned judicial document is processed by the OCR model is taken as an example to illustrate the classification results and reorganization:

[0085] The first text box (marked with O): Applicant for retrial (plaintiff in the first instance, appellant in the second instance): Cai **, male, born on * month * day, 19**, Han nationality, living in ** District, Huaihua City, Hunan Province.

[0086] The second text box (marked with O): Applicant (defendant in the first instance, appellant in the second instance): Huaihua *** Co., Ltd., domiciled at No. ***, Renmin South Road, ** District, Huaihua City, Hunan Province.

[0087] Legal representative: Chen**, general manager of the company.

[0088] The third text box (marked as B): Litigation representative: Nie**, head of the administrator of Huaihua ** Co., Ltd.

[0089] The retrial applicant Cai ** had a property rights confirmation dispute with Huaihua ** Co., Ltd. (hereinafter referred to as ** Company);

[0090] The fourth text box (marked as I): The case was filed by the High People's Court of Hunan Province (2020) Hunan Minzhong No. **, and the applicant applied to this court for a retrial. This court formed a collegial panel in accordance with the law to conduct a review, and the review has now ended.

[0091] When reorganizing according to the positional relationship of the text boxes, since the first and second text boxes are marked as O, their contents can be taken out separately without any ambiguity; the third and fourth text boxes are marked as B and I respectively, and the third and fourth text boxes are in a front-to-back relationship, so the contents of the third and fourth text boxes need to be combined to have the correct paragraph structure.

[0092] See also Figure 4 In one embodiment, step S201 includes:

[0093] S301: Arranging the text content of each text box in the order of words to obtain a text sequence;

[0094] S302: Encode each word in the text sequence through a Bert model to obtain a vector code of each word;

[0095] S303: compressing and preventing overfitting of vector codes of all words in the text sequence through a pooling layer to obtain sentence code data of each text sequence;

[0096] S304: extracting features from the image data by an image feature extractor to obtain an image feature vector for each image data;

[0097] S305: performing attention calculation on the sentence encoding data of each of the text sequences and all of the image feature vectors respectively, and fusing the image feature vectors to obtain a first fused feature vector;

[0098] S306: Fusing each of the sentence encoding data with the first fused feature vector obtained by fusing the sentence encoding data to obtain a second fused feature vector.

[0099] In this embodiment, after arranging the text contents in the text box in sequence, the characters are processed through the Bert model to obtain vector codes of the characters; the vector codes are compressed and anti-overfitting processed before fusion through the pooling layer to facilitate subsequent data processing; after the image data is feature extracted by the image feature extractor, the sentence encoding data of each of the text sequences is respectively calculated with all the image feature vectors for attention, and the image feature vectors are fused to obtain a first fused feature vector after preliminary fusion processing; each of the sentence encoding data is fused with the first fused feature vector obtained by fusion of the sentence encoding data to obtain a second fused feature vector after further fusion.

[0100] By integrating the image features into the text feature vector (sentence encoding data), we can utilize the image position features of the text in the scanned document to more accurately characterize the features of the text in a specific scene, which helps to improve the accuracy of paragraph structure restoration.

[0101] Among them, in step S302, the Bert model is a pre-trained language model based on a bidirectional Transformer (a feature extractor), which is trained through a large-scale training corpus. The vector obtained after the text is encoded by the Bert model has rich representation capabilities, and words with similar semantics are often close in the vector space. Therefore, using the pre-trained Bert model to encode the text sequence in the text box can effectively improve the accuracy of text representation.

[0102] In one embodiment, step S303 includes:

[0103] The sentence encoding data is calculated according to the following formula:

[0104]

[0105] Among them, Bert(w j ) represents the vector encoding obtained after the j-th word in each of the text sequences is encoded by the Bert model.

[0106] In one embodiment, step S305 includes:

[0107] The attention value between the sentence encoding data of each text sequence and all the image feature vectors is calculated according to the following formula:

[0108]

[0109] where e ij The calculation formula is:

[0110] e ij =bv i T W a v j ;

[0111] The first fused feature vector obtained by fusing the image feature vector with the attention value is calculated according to the following formula:

[0112]

[0113] Among them, W a represents the trainable parameter matrix, v j The image feature vector representing the jth image data, bv i Represents sentence encoding data of the i-th text sequence.

[0114] In one embodiment, step S306 includes:

[0115] The second fused feature vector obtained by fusing the sentence coding data with the first fused feature vector is calculated according to the following formula:

[0116] s i =tanh(W b [bv i ; pv i ]+b b );

[0117] Among them, bv i Represents the sentence encoding data of the i-th text sequence, W b represents the trainable parameter matrix, b b is the bias term, [bv i ; pv i ] is a vector concatenation operation.

[0118] In one embodiment, step S204 includes:

[0119] The classification probability of each text box is calculated according to the following formula:

[0120]

[0121] The calculation formula of s(x,y) is:

[0122]

[0123] in, Represents a trainable state transfer matrix, which is in the CRF structure and represents the transition from state y i Transfer to state y i+1 probability; represents the emission matrix, which is the result of normalized mapping in step S203.

[0124] Specifically, in one embodiment, step S203 includes:

[0125] The normalized mapping result is calculated according to the following formula:

[0126] z i =softmax(c i W1+b1);

[0127] where c i The calculation formula is:

[0128]

[0129] Among them, W1 is the trainable parameter matrix, b1 is the bias term; represents the output after forward LSTM encoding, represents the output after backward LSTM encoding, c i is the result of concatenating two vectors; z i is the result after mapping, indicating the probability value of the i-th text box belonging to B, I, or O.

[0130] Specifically, Figure 1 p1~p n are the image data of n text boxes, v1~v n are the image feature vectors corresponding to the image data, w1~w n are n characters in a text box, s1~s n They are respectively the fused sentence encoding data (i.e., the second fused feature vector). The calculation process in the recurrent neural network is common knowledge in the art, so it will not be repeated here.

[0131] See also Figure 5 , a multi-modal scanned document paragraph structure restoration device 400, comprising:

[0132] The OCR recognition unit 401 is used to recognize the text scan data through the pre-trained OCR model to obtain a plurality of text boxes, the text content of each of the text boxes and the coordinate position of each of the text boxes corresponding to the text scan data;

[0133] The optimization and labeling unit 402 is used to remove the text boxes without text content, and arrange the text boxes in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and label each of the text boxes;

[0134] An image acquisition unit 403 is used to extract the picture area where the corresponding text box is located according to each coordinate position, and obtain image data corresponding to each text box;

[0135] A prediction and classification unit 404 is used to perform encoding and fusion processing on the text content and the image data through a sequence classification model, and predict the classification result of each text box;

[0136] The reorganization unit 405 is used to reorganize the paragraphs according to the classification result.

[0137] In one embodiment, the prediction classification unit 404 includes:

[0138] A sentence coding data fusion unit, used for coding and fusing the text content of each text box with the image data to obtain sentence coding data corresponding to the fused text box;

[0139] A sentence coding sorting unit, used for combining each of the sentence coding data in order to obtain a sentence coding sequence;

[0140] A recurrent neural processing unit, used for encoding the sentence encoding sequence through a recurrent neural network, and performing normalized mapping on the sentence encoding sequence after encoding;

[0141] The classification unit is used to perform probability prediction on the normalized sentence encoding sequence through a conditional random field, then decode it through a beam search, and classify each of the text boxes to obtain a classification result.

[0142] In one embodiment, the sentence encoding data fusion unit includes:

[0143] A text sorting unit, used for arranging the text content of each text box in the order of words to obtain a text sequence;

[0144] A Bert vector encoding unit, used to encode each word in the text sequence through a Bert model to obtain a vector encoding of each word;

[0145] A sentence encoding unit, used for compressing and preventing overfitting the vector encodings of all words in the text sequence through a pooling layer to obtain sentence encoding data of each text sequence;

[0146] An image feature extraction unit, used to extract features from the image data through an image feature extractor to obtain an image feature vector for each of the image data;

[0147] A first fusion unit is used to perform attention calculation on the sentence encoding data of each of the text sequences and all of the image feature vectors, and fuse the image feature vectors to obtain a first fused feature vector;

[0148] The second fusion unit is used to fuse each of the sentence encoding data with the first fused feature vector obtained by fusing the sentence encoding data to obtain a second fused feature vector.

[0149] In one embodiment, the first fusion unit comprises:

[0150] The attention calculation unit is used to calculate the attention value between the sentence encoding data of each text sequence and all the image feature vectors according to the following formula:

[0151]

[0152] where e ij The calculation formula is:

[0153] e ij =bv i T W a v j ;

[0154] The attention image fusion unit is used to calculate the first fused feature vector obtained by fusing the image feature vector with the attention value according to the following formula:

[0155]

[0156] Among them, W a represents the trainable parameter matrix, v j The image feature vector representing the jth image data, bv i Represents sentence encoding data of the i-th text sequence.

[0157] In one embodiment, the second fusion unit comprises:

[0158] The sentence coding fusion unit is used to calculate the second fused feature vector obtained by fusing the sentence coding data with the first fused feature vector according to the following formula:

[0159] s i =tanh(W b [bv i ; pv i ]+b b);

[0160] Among them, bv i Represents the sentence encoding data of the i-th text sequence, W b represents the trainable parameter matrix, b b is the bias term, [bv i ; pv i ] is a vector concatenation operation.

[0161] In one embodiment, the sentence encoding unit includes:

[0162] The sentence coding calculation unit is used to calculate the sentence coding data according to the following formula:

[0163]

[0164] Among them, Bert(w j ) represents the vector encoding obtained after the j-th word in each of the text sequences is encoded by the Bert model.

[0165] In one embodiment, the classification unit includes:

[0166] The probability calculation unit is used to calculate the classification probability of each text box according to the following formula:

[0167]

[0168] The calculation formula of s(x,y) is:

[0169]

[0170] in, represents the state transfer matrix, represents the emission matrix.

[0171] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned multimodal scanned document paragraph structure restoration device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0172] At the same time, the division and connection methods of the various units in the above-mentioned multimodal scanned document paragraph structure restoration device are only used for illustrative purposes. In other embodiments, the multimodal scanned document paragraph structure restoration device can be divided into different units as needed, and the various units in the multimodal scanned document paragraph structure restoration device can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned multimodal scanned document paragraph structure restoration device.

[0173] The above-mentioned apparatus for restoring the paragraph structure of a scanned document based on multimodality can be implemented in the form of a computer program, and the computer program can be run on a computer device as shown in the figure.

[0174] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0175] See also Figure 6 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0176] The non-volatile storage medium can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned method for restoring the paragraph structure of a scanned document based on multimodality.

[0177] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0178] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned multi-modal scanned document paragraph structure restoration method.

[0179] The network interface 505 is used to communicate with other devices over the network. Those skilled in the art will appreciate that the structure shown in the figure is only a block diagram of a portion of the structure related to the present application solution, and does not constitute a limitation on the computer device to which the present application solution is applied. The specific computer device may include Figure 6 For example, in some embodiments, the computer device 500 may include only a memory and a processor 502. In such embodiments, the structure and function of the memory and the processor 502 are similar to those of the embodiment of the present invention. Figure 6 The embodiments shown are consistent and will not be described again here.

[0180] The processor 502 is used to run a computer program 5032 stored in a memory to implement the following steps: identify text scan data through a pre-trained OCR model to obtain a number of text boxes, the text content of each text box and the coordinate position of each text box corresponding to the text scan data; remove the text boxes without text content, and arrange them in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and mark each text box; take out the corresponding image area where the text box is located according to each coordinate position, and obtain the image data corresponding to each text box; encode and fuse the text content and the image data through a sequence classification model, and predict the classification result of each text box; and reorganize the paragraphs according to the classification result.

[0181] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0182] Those skilled in the art can understand that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program 5032, and the computer program 5032 can be stored in a computer-readable storage medium. The computer program 5032 is executed by at least one processor 502 in the computer system to implement the process steps of the embodiment of the above method.

[0183] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium can be a non-volatile computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps: recognizing text scan data through a pre-trained OCR model to obtain a number of text boxes, the text content of each of the text boxes, and the coordinate position of each of the text boxes corresponding to the text scan data; removing the text boxes without text content, and arranging them in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and marking each of the text boxes; taking out the corresponding picture area where the text box is located according to each of the coordinate positions, and obtaining the image data corresponding to each of the text boxes; encoding and fusing the text content and the image data through a sequence classification model, and predicting the classification results of each of the text boxes; and reorganizing the paragraphs according to the classification results.

[0184] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0187] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0189] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0191] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for restoring the paragraph structure of a scanned document based on multimodality, characterized in that: include: Recognize the text scan data through a pre-trained OCR model to obtain a plurality of text boxes, the text content of each of the text boxes, and the coordinate position of each of the text boxes corresponding to the text scan data; The text boxes without text content are removed, and the text boxes are arranged in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and each text box is marked; According to each of the coordinate positions, the image area where the corresponding text box is located is taken out to obtain the image data corresponding to each of the text boxes; Performing encoding fusion processing on the text content and the image data through a sequence classification model, and predicting the classification result of each text box; Reorganize the paragraphs according to the classification results; The encoding and fusion processing of the text content and the image data is performed through the sequence classification model, and the classification result of each text box is predicted, including: encoding and fusion of the text content and the image data of each text box to obtain sentence encoding data corresponding to the fused text box; combining each sentence encoding data in sequence to obtain a sentence encoding sequence; encoding the sentence encoding sequence through a recurrent neural network, and normalizing and mapping the sentence encoding sequence after encoding; probabilistically predicting the normalized sentence encoding sequence through a conditional random field, then decoding through a beam search, and classifying each text box to obtain a classification result; The encoding and fusion of the text content of each text box and the image data to obtain the sentence encoding data corresponding to the fused text box includes: arranging the text content of each text box in the order of words to obtain a text sequence; encoding each word in the text sequence through a Bert model to obtain a vector encoding of each word; compressing and anti-overfitting processing the vector encodings of all words in the text sequence through a pooling layer to obtain sentence encoding data of each text sequence; extracting features from the image data through an image feature extractor to obtain an image feature vector of each image data; performing attention calculation on the sentence encoding data of each text sequence and all the image feature vectors respectively, and fusing the image feature vectors to obtain a first fused feature vector; fusing each sentence encoding data with the first fused feature vector obtained by fusing the sentence encoding data to obtain a second fused feature vector.

2. The method for restoring the paragraph structure of a scanned document based on multimodality according to claim 1, characterized in that: The sentence encoding data of each of the text sequences is respectively subjected to attention calculation with all the image feature vectors, and the image feature vectors are fused to obtain a first fused feature vector, including: The attention value between the sentence encoding data of each text sequence and all the image feature vectors is calculated according to the following formula: where e ij The calculation formula is: The first fused feature vector obtained by fusing the image feature vector with the attention value is calculated according to the following formula: Among them, W a represents the trainable parameter matrix, v j The image feature vector representing the jth image data, bv i Represents sentence encoding data of the i-th text sequence.

3. The method for restoring the paragraph structure of a scanned document based on multimodality according to claim 1, characterized in that: The step of fusing each of the sentence encoding data with a first fused feature vector obtained by fusing the sentence encoding data to obtain a second fused feature vector comprises: The second fused feature vector obtained by fusing the sentence coding data with the first fused feature vector is calculated according to the following formula: S i =tanh(W b [bv i pv i ]+b b ); Among them, bv i Represents the sentence encoding data of the i-th text sequence, W b represents the trainable parameter matrix, b b is the bias term, [bv i ; pv i ] is a vector concatenation operation.

4. The method for restoring the paragraph structure of a scanned document based on multimodality according to claim 1, characterized in that: The vector encoding of all words in the text sequence is compressed and processed to prevent overfitting by the pooling layer to obtain sentence encoding data of each text sequence, including: The sentence encoding data is calculated according to the following formula: Among them, Bert(w j ) represents the vector encoding obtained after the j-th word in each of the text sequences is encoded by the Bert model.

5. The method for restoring the paragraph structure of a scanned document based on multimodality according to claim 1, characterized in that: The normalized sentence encoding sequence is processed by the conditional random field, then decoded by beam search, and each text frame is classified to obtain a classification result, including: The classification probability of each text box is calculated according to the following formula: The calculation formula of s(x,y) is: in, represents the state transfer matrix, represents the emission matrix.

6. A multi-modal scanned document paragraph structure restoration device, characterized in that: include: An OCR recognition unit, used to recognize the text scan data through a pre-trained OCR model, and obtain a plurality of text boxes, the text content of each of the text boxes, and the coordinate position of each of the text boxes corresponding to the text scan data; The optimization and annotation unit is used to remove the text boxes without text content, and arrange the text boxes in sequence according to the coordinate positions of the text boxes to obtain a text box sequence, and annotate each of the text boxes; An image acquisition unit, used for taking out the picture area where the corresponding text box is located according to each coordinate position, and obtaining image data corresponding to each text box; A prediction classification unit, used for performing encoding fusion processing on the text content and the image data through a sequence classification model, and predicting a classification result of each text box; The prediction classification unit includes: a sentence coding data fusion unit, which is used to encode and fuse the text content of each text box and the image data to obtain the sentence coding data corresponding to the fused text box; a sentence coding sorting unit, which is used to combine each sentence coding data in sequence to obtain a sentence coding sequence; a recurrent neural processing unit, which is used to encode the sentence coding sequence through a recurrent neural network and normalize and map the sentence coding sequence after encoding; a classification unit, which is used to perform probability prediction on the normalized sentence coding sequence through a conditional random field, then decode it through a beam search, and classify each text box to obtain a classification result; The sentence encoding data fusion unit includes: a text sorting unit, which is used to arrange the text content of each text box in the order of words to obtain a text sequence; a Bert vector encoding unit, which is used to encode each word in the text sequence through a Bert model to obtain a vector encoding of each word; a sentence encoding unit, which is used to compress and prevent overfitting of the vector encodings of all words in the text sequence through a pooling layer to obtain sentence encoding data of each text sequence; an image feature extraction unit, which is used to extract features of the image data through an image feature extractor to obtain an image feature vector of each image data; a first fusion unit, which is used to perform attention calculation on the sentence encoding data of each text sequence and all the image feature vectors respectively, and fuse the image feature vectors to obtain a first fused feature vector; a second fusion unit, which is used to fuse each sentence encoding data with the first fused feature vector obtained by fusing the sentence encoding data to obtain a second fused feature vector; A reorganization unit is used to reorganize the paragraphs according to the classification result.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for restoring the paragraph structure of a scanned document based on multimodality as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the multimodal scanned document paragraph structure restoration method as described in any one of claims 1 to 5.

Images