Reading sequence prediction method and device, electronic equipment and storage medium

Abstract

The invention provides a reading sequence prediction method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of machine learning. The method comprises the following steps: acquiring a to-be-predicted document image; obtaining word information of each word in the to-be-predicted document image; obtaining text block information of each text block in the to-be-predicted document image; predicting a word reading sequence in each text block and a text block reading sequence among the text blocks based on the word information and the text block information; and predicting the reading sequence of the to-be-predicted document image based on the text block reading sequence between the text blocks and the word reading sequence in each text block. According to the method and the device for predicting the reading sequence of the document, under the condition that an optical character recognition (OCR) cannot accurately detect a text line and only a single character can be detected and recognized, the reading sequence prediction of the document with complicated typesetting is realized, and the accuracy of predicting the reading sequence of the document is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a reading order prediction method, device, electronic device and computer-readable storage medium. Background Art

[0002] The purpose of document image recognition is to obtain text information in an image. By recognizing text in an image, it helps to manage and use the data in the image. For simple documents, you only need to use optical character recognition (OCR) detection and recognition to obtain the text coordinates and corresponding text content of each line, and connect all the content in order from top to bottom and from left to right to get the result of document image recognition. If the layout of the document is more complicated, you need to determine the reading order of the document.

[0003] However, in some scenarios, the document image to be predicted may have text distortion, tilt, twist, etc., which may cause OCR to be unable to accurately detect text lines and can only accurately detect and recognize single characters. At this time, if the reading order of the document image is predicted by the method of predicting the reading order with text blocks as the basic units in the related technology, even if the text blocks are obtained, the words in the same text line may be misplaced. After determining the reading order of the text block, the reading order between the words in the text block cannot be determined. Therefore, only a reading order prediction scheme with words as the basic units can be used. However, for documents with complex typesetting, the prediction accuracy obtained by the reading order prediction scheme with words as the basic units is low. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a reading order prediction method, device, electronic device and computer-readable storage medium to solve the problem that when the document image to be predicted has text distortion, tilt, or twist, the method of predicting the reading order using text blocks as basic units cannot determine the reading order between each word in the text block, and the reading order prediction scheme using words as basic units has a low prediction accuracy.

[0005] According to a first aspect of the present disclosure, a reading order prediction method is provided, comprising: obtaining a document image to be predicted; obtaining word information of each word in the document image to be predicted; obtaining text block information of each text block in the document image to be predicted; predicting a word reading order within each of the text blocks and a text block reading order between the text blocks based on the word information and the text block information; predicting a reading order of the document image to be predicted based on the text block reading order between the text blocks and a word reading order within each of the text blocks.

[0006] According to a second aspect of the present disclosure, a reading order prediction device is provided, comprising: an image acquisition module for acquiring a document image to be predicted; an information extraction module for acquiring word information of each word in the document image to be predicted; the information extraction module is also used to acquire text block information of each text block in the document image to be predicted; a sequence prediction module is used to predict a word reading order within each of the text blocks and a text block reading order between the text blocks based on the word information and the text block information; the sequence prediction module is also used to predict a reading order of the document image to be predicted based on a text block reading order between the text blocks and a word reading order within each of the text blocks.

[0007] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the above-mentioned reading order prediction method.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned reading order prediction method.

[0009] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0010] In the reading order prediction method provided in the exemplary embodiment of the present disclosure, a document image to be predicted is obtained; word information of each word in the document image to be predicted is obtained; text block information of each text block in the document image to be predicted is obtained; the reading order of words in each text block and the reading order of text blocks between text blocks are predicted based on the word information and text block information; the reading order of the document image to be predicted is predicted based on the reading order of text blocks between text blocks and the reading order of words in each text block. The embodiment of the present disclosure obtains word information and text block information in the document image to be detected, and determines the reading order between text blocks based on the word information and text block information, so as to predict the reading order of the entire document image to be predicted based on the reading order of words in each text block. By combining the reading order prediction based on text blocks as basic units and the reading order prediction based on words as basic units, the reading order prediction of documents with complex layout can be realized when OCR cannot accurately detect text lines and can only detect and recognize single characters, thereby improving the accuracy of document reading order prediction.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A flowchart of a reading order prediction method according to an exemplary embodiment of the present disclosure is shown;

[0014] Figure 2 A schematic diagram of a document image to be predicted according to an exemplary embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram showing another document image to be predicted according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 4 A flowchart of a method for obtaining word information of each word in a document image to be predicted in a reading order prediction method according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 5 A flow chart of a method for training a layout analysis model in a reading order prediction method according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic block diagram of a reading order prediction device according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0022] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The following describes the aspects of the present disclosure with reference to the accompanying drawings for explanation.

[0026] The present application embodiment first provides a reading order prediction method. Figure 1 As shown, the reading order prediction method specifically includes the following steps:

[0027] S110: Acquire a document image to be predicted;

[0028] S120: Obtain word information of each word in the document image to be predicted;

[0029] S130: Acquire text block information of each text block in the document image to be predicted;

[0030] S140: predicting a word reading order within each text block and a text block reading order between text blocks based on the word information and the text block information;

[0031] S150: predicting the reading order of the document image to be predicted based on the text block reading order between the text blocks and the word reading order within each text block.

[0032] The disclosed embodiment obtains word information and text block information in the document image to be detected, and determines the reading order between text blocks based on the word information and text block information, and can predict the reading order of the entire document image to be predicted based on the reading order of words in each text block. By combining the reading order prediction based on text blocks as basic units with the reading order prediction based on words as basic units, the reading order prediction of documents with complex layout can be realized when OCR cannot accurately detect text lines and can only detect and recognize single characters, thereby improving the accuracy of document reading order prediction.

[0033] Next, in another embodiment, the above steps are described in more detail.

[0034] In step S110, a document image to be predicted is obtained.

[0035] In the embodiment of the present application, the document image to be predicted is a document image for which reading order prediction is required.

[0036] In an embodiment of the present application, the image of the document to be predicted can be collected by an electronic device with a shooting function. For example, a paper document can be photographed by a camera function of a smart phone to obtain the image of the document to be predicted.

[0037] In the embodiment of the present application, after the image of the document to be predicted is acquired through the shooting function of the electronic device, since the captured image may contain a background area, in order to ensure the accuracy of operations such as document detection and layout analysis in the subsequent reading order prediction process, the embodiment of the present application can also detect the image of the document to be predicted from the captured image based on a target detection algorithm after acquiring the captured image. Exemplarily, the target detection algorithm can be a real-time object detection algorithm (You Only Look Once, YOLO), or other image detection algorithms that can achieve the same function, which is not specifically limited in the embodiment of the present application.

[0038] In the embodiment of the present application, the document image to be predicted may include different types of content such as text, title, introduction, picture and table.

[0039] In the embodiment of the present application, when obtaining the image of the document to be predicted, the layout information of the image of the document to be predicted can also be obtained. Exemplarily, the layout information can include layout data, and the layout data can be used to describe the arrangement of each content area in the document to be predicted. Among them, the content area can be different categories of content such as the above-mentioned text, title, introduction, picture and table.

[0040] In the embodiment of the present application, the arrangement of the content area in the above-mentioned document image to be predicted may be in various different situations. For example, the above-mentioned document image to be predicted may be arranged in a simple manner, such as Figure 2 As shown, Figure 2 The document image to be predicted in contains only one column of content; it can also be a complex arrangement, such as Figure 3 As shown, Figure 3 The document image to be predicted in includes multiple categories of content such as title, author, text and picture, and the text is divided into two columns, left and right, and there is also a picture column below the right text column. It should be noted that the above Figure 2 and Figure 3 This is just an exemplary description. The document image to be predicted in the embodiment of the present application may also be arranged in other ways, such as three columns, combined columns, irregular columns, etc.

[0041] For the above Figure 2 For the document image to be predicted with a simple arrangement as shown, when detecting the reading order prediction, it is only necessary to use OCR detection and recognition to obtain the text coordinates and corresponding text content of each line, and then connect all the contents in order from top to bottom and from left to right.

[0042] However, in practical applications, the acquired image of the document to be predicted may be distorted. For example, when the image of the document to be predicted is collected by the above-mentioned electronic device, the image of the document to be predicted may have text tilt, distortion, etc. due to reasons such as the shooting angle. Since the same line of text may be misplaced, the words in the same line may not be correctly detected as a line using text line detection, thereby affecting the accuracy of the reading order prediction. At this time, compared with the reading prediction method using text lines or text blocks as basic units, the reading prediction method using words as basic units is more reliable. However, the reading order prediction scheme using words as basic units is difficult to use in such situations. Figure 3 Better results are achieved on the document image with complex layout shown.

[0043] In order to solve the above problems, after obtaining the document image to be predicted, the embodiment of the present application predicts the reading order of the document image to be predicted through the following steps S120 to S150. This can improve the prediction accuracy of the reading order of the document image to be predicted when the document image to be predicted has a complex layout and / or the text is tilted, distorted, etc.

[0044] In step S120, word information of each word in the document image to be predicted is obtained.

[0045] In the embodiment of the present application, the above words are units constituting the text content in the document image to be predicted.

[0046] In the embodiment of the present application, the above-mentioned words can be single-character Chinese characters, letters, punctuation marks or special symbols, or English words composed of multiple letters, or other character forms that meet the definition of the above-mentioned words. The embodiment of the present application does not make any special limitations on this.

[0047] In the embodiment of the present application, the word information may include word position information and word text, wherein the word position information is used to describe the position of the word in the document image to be predicted, and the word text is the text content of the word.

[0048] In the embodiment of the present application, the process of obtaining the word information of each word in the document image to be predicted is exemplarily as follows: Figure 4 As shown, it can be implemented by the following steps S410 to S430:

[0049] S410: Obtain word position information of a word in the document image to be predicted.

[0050] In an embodiment of the present application, the above-mentioned word position information may be coordinate information. Exemplarily, the coordinate information may include the coordinates of four vertices of the detection box corresponding to the word.

[0051] In the embodiment of the present application, for example, the coordinate information of the word can be obtained based on the Detection and Recognition Neural Network (DBNet) algorithm. It should be noted that the above scenario is only an exemplary description, and other methods with the same function can also be used to obtain the word position information, and the embodiment of the present application is not particularly limited by comparison.

[0052] S420: Determine a word screenshot based on the word position information, wherein the word screenshot contains at least one word text.

[0053] In an embodiment of the present application, after the word position information is obtained, the text in the document image to be detected can be intercepted into multiple word screenshots based on the word position information. Exemplarily, after the vertex coordinates of the word detection box are obtained, the position of the corresponding detection box in the document image to be detected is determined based on the vertex coordinates, and the corresponding word screenshots are intercepted.

[0054] In the embodiment of the present application, the above-mentioned word screenshot may contain one word or multiple words, and the embodiment of the present application does not make any special limitation on this.

[0055] S430: Recognize the word text of the word in the word screenshot.

[0056] In the embodiment of the present application, after the word screenshot is captured, the word text of each word in the word screenshot can also be recognized by a visual text recognition algorithm (Scene Text Recognition with a Single Visual Model, SVTR). It should be noted that the above scenario is only an exemplary description, and other methods with the same function can also be used to obtain the word text, and the embodiment of the present application does not make special limitations for comparison.

[0057] In step S130, text block information of each text block in the document image to be predicted is obtained.

[0058] In the embodiment of the present application, the text block is the content area in the document image to be predicted.

[0059] In the embodiment of the present application, the above-mentioned text block information includes text block position information and text block category information of the above-mentioned text block, that is, the position of the above-mentioned content area in the document image to be predicted, and the content type contained in the above-mentioned content area.

[0060] In an embodiment of the present application, the text block position information may be coordinate information. Exemplarily, the coordinate information may include the coordinates of four vertices of the detection box corresponding to the text block.

[0061] In the embodiment of the present application, the above-mentioned text block categories can be titles, texts, charts, formulas and the like, and the embodiment of the present application does not make any special limitation on this.

[0062] In an embodiment of the present application, the text block information of each text block in the above-mentioned document image to be predicted can be obtained by performing layout analysis on the document image to be predicted. Exemplarily, the process of obtaining the text block information of each text block in the document image to be predicted can be implemented as follows: input the document image to be predicted into a trained layout analysis model to obtain the text block position information and text block category information of each text block in the document image to be predicted; wherein the layout analysis model is used to output the layout information of the document image to be predicted, and the layout information includes the text block position information and the text block text information.

[0063] In the embodiment of the present application, the training of the layout analysis model is as follows: Figure 5 As shown, it can be implemented by the following steps S510 to S530:

[0064] S510: Acquire a first training image set.

[0065] In an embodiment of the present application, the above-mentioned layout analysis model can be obtained based on a document understanding multimodal pre-trained model (LayoutLM version 3, LayoutLMv3). LayoutLMv3 combines text classification and layout analysis techniques to jointly model and predict the text and layout information in the document image to be predicted. By learning the correlation between text and layout, the input text can be analyzed to obtain the layout information of the text.

[0066] In the embodiment of the present application, the above LayoutLMv3 includes two stages: pre-training and fine-tuning. The above layout analysis module is obtained by fine-tuning the pre-training model obtained in the pre-training stage.

[0067] In an embodiment of the present application, in the above-mentioned pre-training stage, LayoutLMv3 pre-trains the model through large-scale unlabeled text and layout data to learn the correlation between text and layout. Specifically, in the pre-training stage, LayoutLMv3 can adopt a multi-task learning framework to simultaneously predict text classification, layout structure and position information, thereby capturing the complex relationship between text and layout. This multi-task learning method enables LayoutLMv3 to understand the structure and meaning of the document more comprehensively. Specifically, the principle of the LayoutLMv3 pre-training stage is as follows: obtain the text position and information through OCR, and obtain the image information of the image patch (Image Patches) corresponding to the text; obtain the alignment information of the text and the image through word encoding (word embedding) and image patch encoding (patch embedding); input the multimodal information into the sequence model (Transformer) structure based on the attention mechanism for multi-task pre-training.

[0068] In the embodiment of the present application, in the fine-tuning stage, LayoutLMv3 uses a labeled data set to further train the pre-trained model so that the obtained model can adapt to the needs of specific tasks. For example, the model can be fine-tuned using a labeled document data set to classify documents, extract key information, or perform other related tasks; or, in the embodiment of the present application, the above-mentioned layout analysis model can be obtained by fine-tuning the pre-trained model, so as to achieve the task of performing layout analysis on the document to be predicted.

[0069] In the embodiment of the present application, the first training image set is a labeled training data set used in the fine-tuning stage.

[0070] S520: marking text block position information and text block category information of the text blocks included in each first training image in the first training image set;

[0071] In an embodiment of the present application, the label of the first training image set can be generated by annotating the first training image set, and the pre-trained model obtained above can be fine-tuned using the first training image set with the label.

[0072] In an embodiment of the present application, the label generated by manual annotation includes the text block position information and text block category information.

[0073] S530: Using the first training image as training data, and using the text block position information and text block category information of each text block as a label, to train and obtain a layout analysis model.

[0074] In the embodiment of the present application, the first training image is used as training data, and the text block position information and text block category information are used as labels, and the pre-trained model obtained above is further trained to obtain the layout analysis model. Subsequently, by inputting the document image to be predicted into the layout analysis model, the layout information in the document image to be predicted can be obtained through the layout analysis model, and the layout information includes the position information and category information of each text block in the document image to be predicted.

[0075] In an embodiment of the present application, the pre-trained model of the above-mentioned LayoutLMv3 can be fine-tuned to obtain a layout analysis model. The layout analysis based on LayoutLMv3, mainly with the help of LayoutLMv3 containing pre-trained weights, is fine-tuned on the layout analysis data set of the embodiment of the present application (that is, the above-mentioned first training image set), so as to achieve accurate parsing and understanding of the document layout structure. By converting the PDF document into a picture format and using rectangular frames to divide the various parts of the document, key information in the document, such as titles, text, charts, formulas, etc., can be extracted. At the same time, by predicting whether the corresponding image patches of text words are blocked, LayoutLMv3 can also learn cross-modal alignment, further improving the accuracy of layout analysis.

[0076] In an embodiment of the present application, in the pre-training stage based on LayoutLMv3, by inputting multimodal information into a sequence model (Transformer) structure based on an attention mechanism, multi-task pre-training is performed to obtain the above-mentioned pre-training model. After obtaining the pre-training model, in the fine-tuning stage of step S530, only the image information of the first training image needs to be input, and the pre-training model is supervised and trained through the pre-annotated information in the first training image set, and the weight of the pre-training model is fine-tuned to obtain the above-mentioned layout analysis model.

[0077] In step S140 , a word reading order within each text block and a text block reading order between text blocks are predicted based on the word information and the text block information.

[0078] In an embodiment of the present application, after obtaining the above-mentioned word information and text block information, the text block reading order and the word reading order can be predicted based on the word information and the text block information. Exemplarily, the process can be implemented as follows: determine the words contained in each text block based on the word position information and the text block position information; determine the source text block sequence of each text block, and the source word sequence of the words in each text block, wherein the source text block sequence is a sequence obtained by sorting the text blocks according to the default order, and the source word sequence is a sequence obtained by sorting the words in the text blocks according to the default order; for each text block, input the source sequence of the corresponding words into the trained first order prediction model to obtain the word reading order in each text block; input the source text block sequence into the trained second order prediction model to obtain the text block reading order between text blocks.

[0079] In the embodiment of the present application, the above default order can be determined based on human reading habits. For example, the default order can be a reading order of text lines from top to bottom and each line of text from left to right.

[0080] In an embodiment of the present application, the above-mentioned determination of the words contained in each text block based on the word position information and the text block position information can be specifically implemented as follows: after obtaining the vertex coordinates of the word and text block detection frames, the degree of overlap of the word detection frame and the text block detection frame is determined based on the vertex coordinates of the word detection frame and the text block detection frame; when the degree of overlap is greater than a preset threshold, the corresponding word screenshot is placed in the corresponding text block. Exemplarily, the determination of the degree of overlap of the word detection frame and the text box detection frame can be achieved by calculating the intersection over union (IOU) of the word detection frame and the text box detection frame.

[0081] In an embodiment of the present application, the training of the above-mentioned first order prediction model can be achieved through the following process: obtaining a second training image set; the second training image set includes multiple second training images; determining the word position information of each word in each second training image, and determining the source word sequence of the words in the second training image based on the word position information; determining the target word sequence of the words in the second training image; based on the source word sequence and the target word sequence of the words, training to obtain the first order prediction model; wherein the first order prediction model is used to output the target word sequence corresponding to the input image of the first order prediction model.

[0082] In an embodiment of the present application, the above-mentioned first order prediction model can be trained based on a pre-trained reading order detection model (LayoutReader) of text and layout. The input of the LayoutReader model is the source sequence spliced ​​with the target sequence, and the output is the predicted sequence corresponding to the input sequence position. Specifically, the above-mentioned training process can be implemented as follows: take the coordinates and text of all words as the two dimensions of the training data, splice all the words on the current page in the default order from top to bottom and from left to right to form a source sequence, and splice all the words on the current page in their correct order to form a target sequence; the above-mentioned first order prediction model is obtained by training based on the above-mentioned source sequence and target sequence. It should be noted that the above-mentioned scenario is only an exemplary description. In addition to LayoutReader, the embodiment of the present application can also use other methods to train the above-mentioned first order prediction model, for example, a method based on geometric rules, etc., and the embodiments of the present disclosure do not make special limitations on this.

[0083] In an embodiment of the present application, the word coordinates and word contents in the text block are spliced ​​together based on the organization of the source sequence and the target sequence, and the first order prediction model obtained by LayoutReader training is input to obtain the output sequence, and the word contents of the output sequence are spliced ​​together, which is the word reading order in the corresponding text block predicted by the first order prediction model.

[0084] In an embodiment of the present application, the training of the above-mentioned second sequential prediction model can be achieved through the following process: obtaining a third training image set; the third training image set includes multiple third training images, and the third training image includes multiple text blocks; determining the text block position information of each text block in each third training image, and determining the source text block sequence of the text block in the third training image based on the text block position information; determining the target text block sequence of the text block in the third training image; based on the source text block sequence and the target text block sequence of the text block, training to obtain the second sequential prediction model; wherein the second sequential prediction model is used to output the target text block sequence corresponding to the input image of the second sequential prediction model.

[0085] In an embodiment of the present application, the above-mentioned second order prediction model can also be obtained based on LayoutReader training. Then, in the process of predicting the reading order of text blocks, the input of the above-mentioned second order prediction model is the coordinates of all text blocks in the document image page to be predicted and the correctly sorted text in the text block, and the reading order between text blocks can be obtained according to the output sequence of the second order prediction model. The process of training the second order prediction model based on LayoutReader is the same as the principle of training the first prediction model, and will not be repeated here. It should be noted that the above scenario is only an exemplary description. In addition to LayoutReader, the embodiment of the present application can also use other methods to train the above-mentioned second order prediction model, for example, a method based on geometric rules, etc., and the embodiment of the present disclosure does not make special limitations on this.

[0086] In step S150 , the reading order of the document image to be predicted is predicted based on the text block reading order between the text blocks and the word reading order within each text block.

[0087] In the embodiment of the present application, after the reading order of the text blocks and the reading order of the words in each text block are predicted, the reading order of the entire document image to be predicted can be obtained based on the reading order of the text blocks and the reading order of the words in each text block. For example, the order of reading the text blocks can be determined based on the reading order of the text blocks, and then in the determined text blocks, the text information of the document image to be predicted can be obtained based on the reading order of the words.

[0088] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0089] The embodiment of the present application can divide the reading order prediction device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or software functional modules.

[0090] The present application embodiment provides a reading order prediction device. Figure 6As shown, the reading order prediction device 600 may include an image acquisition module 610, an information extraction module 620 and a sequence prediction module 630, wherein:

[0091] An image acquisition module 610 is used to acquire a document image to be predicted;

[0092] An information extraction module 620 is used to obtain word information of each word in the document image to be predicted;

[0093] The information extraction module 620 is also used to obtain text block information of each text block in the document image to be predicted;

[0094] A sequence prediction module 630, for predicting a word reading order within each text block and a text block reading order between text blocks based on word information and text block information;

[0095] The sequence prediction module 630 is further used to predict the reading order of the document image to be predicted based on the text block reading order between the text blocks and the word reading order within each text block.

[0096] In an embodiment of the present application, the above-mentioned word information includes word position information and word text, and the above-mentioned information extraction module is specifically used to: obtain the word position information of the word in the document image to be predicted; determine a word screenshot based on the word position information, and the word screenshot contains at least one word text; recognize the word text in the word screenshot.

[0097] In an embodiment of the present application, the above-mentioned text block information includes text block position information and text block category information, and the above-mentioned information extraction module is specifically used to: input the document image to be predicted into a trained layout analysis model to obtain the text block position information and text block category information of each text block in the document image to be predicted; wherein the layout analysis model is used to output the layout information of the document image to be predicted, and the layout information includes text block position information and text block text information.

[0098] In an embodiment of the present application, the above-mentioned reading order prediction device also includes a model training module, which is specifically used to: obtain a first training image set; annotate the text block position information and text block category information of the text blocks contained in each first training image in the first training image set; use the first training image as training data, and use the text block position information and text block category information of each text block as labels to train a layout analysis model.

[0099] In an embodiment of the present application, the above-mentioned sequence prediction module is specifically used to: determine the words contained in each text block based on word position information and text block position information; determine the source text block sequence of each text block, and the source word sequence of the words in each text block, wherein the source text block sequence is a sequence obtained by sorting the text blocks according to a default order, and the source word sequence is a sequence obtained by sorting the words in the text blocks according to a default order; for each text block, the source sequence of the corresponding words is input into a trained first sequence prediction model to obtain the word reading order in each text block; the source text block sequence is input into a trained second sequence prediction model to obtain the text block reading order between text blocks.

[0100] In an embodiment of the present application, the above-mentioned model training module is specifically used to: obtain a second training image set; the second training image set includes multiple second training images; determine the word position information of each word in each second training image, and determine the source word sequence of the words in the second training image based on the word position information; determine the target word sequence of the words in the second training image; based on the source word sequence and target word sequence of the words, train to obtain a first order prediction model; wherein the first order prediction model is used to output the target word sequence corresponding to the input image of the first order prediction model.

[0101] In an embodiment of the present application, the above-mentioned model training module is specifically used to: obtain a third training image set; the third training image set includes multiple third training images, and the third training image includes multiple text blocks; determine the text block position information of each text block in each third training image, and determine the source text block sequence of the text block in the third training image based on the text block position information; determine the target text block sequence of the text block in the third training image; based on the source text block sequence and the target text block sequence of the text block, train to obtain a second sequential prediction model; wherein the second sequential prediction model is used to output the target text block sequence corresponding to the input image of the second sequential prediction model.

[0102] In addition, the specific implementation details of the above-mentioned reading order prediction device have been described in detail in the corresponding position of the reading order prediction method, so they will not be repeated here.

[0103] Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. Figure 7 , which shows a structural schematic diagram of an electronic device 700 suitable for implementing the embodiments of the present application. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703 to implement the voice control method of the embodiment described in the present disclosure. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0105] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart, thereby implementing the voice control method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0107] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0109] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0110] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0111] Obtaining a document image to be predicted;

[0112] Obtain word information of each word in the document image to be predicted;

[0113] Obtaining text block information of each text block in the document image to be predicted;

[0114] Predicting a word reading order within each text block and a text block reading order between text blocks based on word information and text block information;

[0115] The reading order of the document image to be predicted is predicted based on the text block reading order between the text blocks and the word reading order within each text block.

[0116] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0117] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0120] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0123] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0124] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A reading order prediction method, characterized in that: include: Obtaining a document image to be predicted; Obtaining word information of each word in the document image to be predicted; Acquiring text block information of each text block in the document image to be predicted; Predicting a word reading order within each of the text blocks and a text block reading order between the text blocks based on the word information and the text block information; The reading order of the document image to be predicted is predicted based on the text block reading order between the text blocks and the word reading order within each of the text blocks.

2. The reading order prediction method according to claim 1, characterized in that: The word information includes word position information and word text, and the acquiring the word information of each word in the document image to be predicted includes: Obtaining the word position information of the word in the document image to be predicted; Determine a word screenshot based on the word position information, wherein the word screenshot contains at least one word text; The word text in the word screenshot is recognized.

3. The reading order prediction method according to claim 2, characterized in that: The text block information includes text block position information and text block category information; the step of obtaining the text block information of each text block in the document image to be predicted includes: The document image to be predicted is input into a trained layout analysis model to obtain the text block position information and the text block category information of each text block in the document image to be predicted; wherein the layout analysis model is used to output the layout information of the document image to be predicted, and the layout information includes the text block position information and the text block text information.

4. The reading order prediction method according to claim 3, characterized in that: The method further comprises: Obtaining a first training image set; marking the text block position information and the text block category information of the text block contained in each first training image in the first training image set; The first training image is used as training data, and the text block position information and the text block category information of each text block are used as labels to train and obtain the layout analysis model.

5. The reading order prediction method according to claim 3, characterized in that: The predicting of the word reading order in each of the text blocks and the text block reading order between the text blocks based on the word information and the text block information includes: Determine the words contained in each of the text blocks based on the word position information and the text block position information; Determine a source text block sequence of each of the text blocks, and a source word sequence of the words in each of the text blocks, wherein the source text block sequence is a sequence obtained by sorting the text blocks according to a default order, and the source word sequence is a sequence obtained by sorting the words in the text blocks according to a default order; For each of the text blocks, the source sequence of the corresponding words is input into a trained first order prediction model to obtain a reading order of the words in each of the text blocks; The source text block sequence is input into a trained second sequence prediction model to obtain the text block reading order between the text blocks.

6. The reading order prediction method according to claim 5, characterized in that: The method further comprises: Acquire a second training image set; the second training image set includes a plurality of second training images; Determining word position information of each of the words in each of the second training images, and determining the source word sequence of the words in the second training image based on the word position information; determining a target word sequence of the words in the second training image; Based on the source word sequence and the target word sequence of the words, the first sequence prediction model is trained to obtain the first sequence prediction model; wherein the first sequence prediction model is used to output the target word sequence corresponding to the input image of the first sequence prediction model.

7. The reading order prediction method according to claim 5, characterized in that: The method further comprises: Acquire a third training image set; the third training image set includes a plurality of third training images, and the third training image includes a plurality of the text blocks; Determining text block position information of each of the text blocks in each third training image, and determining the source text block sequence of the text blocks in the third training image based on the text block position information; determining a target text block sequence of the text blocks in the third training image; Based on the source text block sequence and the target text block sequence of the text block, the second sequential prediction model is trained; wherein the second sequential prediction model is used to output the target text block sequence corresponding to the input image of the second sequential prediction model.

8. A reading order prediction device, characterized in that: include: An image acquisition module, used to acquire the image of the document to be predicted; An information extraction module, used for obtaining word information of each word in the document image to be predicted; The information extraction module is further used to obtain text block information of each text block in the document image to be predicted; A sequence prediction module, used for predicting a word reading order within each of the text blocks and a text block reading order between the text blocks based on the word information and the text block information; The sequence prediction module is further used to predict the reading order of the document image to be predicted based on the text block reading order between the text blocks and the word reading order within each text block.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images